Where Airflow Fits
4 systems, one nightly report
Follow the nightly report end to end to see where the line between what Airflow owns and what its systems own falls. Bookings and reviews are written by the app into a transactional database. The exports land as files in object storage. A warehouse loads the files and runs the SQL that aggregates them into a daily table. A dashboard tool reads that table at 09:00. Each of those 4 systems stores data or computes on it. Every arrow between them is a command an Airflow task issues.
- App database (source, transactional store)
- Export tasks (transform, Airflow)
- Object storage (storage, landing files)
- Load and aggregate (transform, Airflow)
- Warehouse (storage, daily tables)
- Dashboard (consumer, BI tool)
- App database -> Export tasks
- Export tasks -> Object storage
- Object storage -> Load and aggregate
- Load and aggregate -> Warehouse
- Warehouse -> Dashboard
- Sources: App database
- Landing: Export tasks, Object storage
- Warehouse: Load and aggregate, Warehouse
- Serving: Dashboard
| Step | System that does the work | What Airflow does |
|---|---|---|
| Export yesterday's bookings | App database, via a dump command | Starts the command at 00:00, waits, records the outcome |
| Export yesterday's reviews | App database, the same way | Starts it beside the bookings export, since no edge joins them |
| Load both files | Warehouse, via a load statement | Starts it once both exports show success |
| Aggregate into a daily table | Warehouse, via one SQL statement | Sends the statement, waits, records success or failure |
| Refresh the dashboard | Dashboard tool, via its API | Calls the API and records the response |
Airflow's own record
Airflow's own territory is the metadata around the work: the graph of tasks and edges, the schedule, the decision at every moment of which task may start, and the record of every run and every attempt, with its start and end times and final state, plus the log it wrote. That record lives in a database Airflow runs for itself, separate from your warehouse. When a task runs, its command executes on a worker machine, and only its state and timestamps come back into that database, along with its log.
Kept in Airflow's own database
- Every DAG it has read, with its tasks and edges
- Every run, with its scheduled time and its state
- Every task attempt, with start, end, state and a pointer to its log
- Nothing from the tables your tasks read or write
One orchestrator, many teams
A company usually runs one Airflow deployment and gives every team a place in its dags folder. The finance team's month-end DAG, the growth team's hourly attribution DAG and the machine learning team's weekly retrain DAG share one scheduler and one UI, and they write to the same record, while each calls its own systems. That shared record is what makes the orchestrator the place to stand when something is late: one screen lists all 300 DAGs and their latest nights, and the question of whether finance's load finished before growth's join started has an answer there without opening either team's warehouse.
Heavy compute belongs in the warehouse
Storage and heavy compute belong to the systems built for them. A task can pull 40 million rows into Python on an Airflow machine and deduplicate them there, and it will work on the first night, but that compute now runs on the machine Airflow uses to run everything else, and when 2 such tasks overlap the whole deployment slows. The version that scales sends the warehouse one statement, an insert that selects distinct rows, and waits for it: the warehouse does the work it was sized for, and the task's own cost is one open connection. Small glue work fits in a task: reading a few thousand rows to decide a path, renaming a file, calling an API. The test is where the bytes go, and a task whose memory grows with the business is in the wrong place.
- Its memory grows with the business, month after month
- It runs for an hour on the orchestrator's machine while the warehouse sits idle
- Its log is full of row counts rather than one statement and one outcome
- Another team's tasks slow down whenever it runs
- Do
- Give each task one command to one system
- Keep Airflow's state and your business data in separate stores
- Let the warehouse aggregate; let Airflow wait and record
- Size a task's memory to stay flat as the data grows
- Avoid
- Load a full table into Python inside a task
- Treat Airflow's database as a place to keep business rows
- Compute in the task what the warehouse would do in one statement
- Chain systems by clock time outside the graph
Everything the UI shows you in the coming sections is drawn from Airflow's own record, and nothing in it is a row of yours. When a run looks wrong, the record tells you which task ended how and when, and the data itself has to be inspected in the system that holds it.
How a task opens a connection to a database or an API, and how it hands a value to the task after it, are covered in the Tasks and Data track.
About this lesson
This section is part of the DAG Anatomy: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.