Skip to main content

What is a DAG

What is a DAG?

Airflow refers to what we've been calling "pipelines" as DAGs (directed acyclic graphs). In computer science, a directed acyclic graph simply means a workflow which only flows in a single direction. Each "step" in the workflow (an edge) is reached via the previous step in the workflow until we reach the beginning. The connection of edges is called a vertex.
If this remains unclear, consider how nodes in a tree data structure relate to one another. Every node has a "parent" node, which of course means that a child node cannot be its parents' parent. That's it - there's no need for fancy language here.
Edges in a DAG can have numerous "child" edges. Interestingly, a "child" edge can also have multiple parents (this is where our tree analogy fails us). Here's an example:

An example DAG structure.

In the above example, the DAG begins with edges 1, 2 and 3 kicking things off. At various points in the pipeline, information is consolidated or broken out. Eventually, the DAG ends with edge 8.
Labels:

Comments

Post a Comment

Popular posts from this blog

Install pip on Windows

If you haven't downloaded get-pip.py yet: You need to download this file from the official pip website or use curl in your command prompt. Using curl (recommended): Open Command Prompt and type: DOS curl https://bootstrap.pypa.io/get-pip.py -o get-pip.py This will download the file to your current directory in the Command Prompt. Manual Download: Go to https://bootstrap.pypa.io/get-pip.py in your web browser, right-click on the page, select "Save as...", and save the file as get-pip.py to a location you can easily navigate to (e.g., your Downloads folder, or a dedicated Python projects folder). The file is in a different directory: You might have downloaded get-pip.py to your Downloads folder, your Desktop, or another location, but you're trying to run the command from C:\Users\gparn . Solution: Navigate to the directory where you saved get-pip.py in your Command Prompt using the cd (change directory) command. Example (if in Downloads): DOS cd C:\User...
Post a Comment
Read more