What Airflow Is For
3 scripts and a timer
A marketplace team needs a report of yesterday's bookings and reviews on a dashboard by 09:00. The first version is 3 scripts on a server's timer: export_bookings.py at 01:00, export_reviews.py at 01:15, and build_report.py at 02:00. The 02:00 slot is a guess that the 2 exports take under 45 minutes. One night in December the bookings export runs for 70 minutes because a holiday weekend tripled the rows. build_report.py starts at 02:00 on schedule, reads a half-written bookings file, and finishes with no error. The dashboard shows a 40% drop in bookings at 09:00, and the on-call engineer spends the morning finding out why.
Moving the report to 03:00 buys a month. The timer still knows nothing about the exports: it fires at a clock time whether the work it depends on has finished. Every question the engineer asks that morning comes down to whether one step had finished before the next began, and the timer records none of it.
| Question that morning | What the timer can tell you |
|---|---|
| Did the bookings export finish before the report started? | Nothing; each timer fires on its own clock |
| Which night did this start? | Nothing; no record of past starts exists |
| Why did the export take 70 minutes? | Only the lines the script printed, if someone saved them to a file |
| Rerun the report for that one night | Edit a date in the script by hand and run it again |
Airflow decides when work may start
Airflow takes over the decision the timer could never make: whether a piece of work may start yet. You write the 3 steps as tasks and say that build report comes after both export bookings and export reviews. Airflow starts the 2 exports at the scheduled time. It starts the report only once it has seen both finish with success. If the bookings export runs 70 minutes, the report starts at 02:10 instead of reading a half-written file. Airflow records when each attempt starts and ends and whether it succeeded. It keeps the log of every attempt beside that record, and the browser UI draws the whole history.
The rest of this lesson uses 3 words. A DAG is one described workflow: the tasks and the order between them, defined in one Python file. A task is one unit of work in it, such as running a shell command. A run is one execution of the whole DAG for one scheduled time, so the DAG for the nightly report has one run per night.
- get_bookings (transform, Airflow)
- get_reviews (transform, Airflow)
- build_report (transform, Airflow)
- get_bookings -> build_report
- get_reviews -> build_report
Airflow is an orchestrator
Airflow is an orchestrator. It starts work on other systems and waits for the result. Then it records what happened and decides what may start next. The bookings rows live in the application database and the warehouse; the report is computed by the Python script and the SQL it sends; Airflow holds the schedule and the order of tasks, and keeps the history of every run. A task in Airflow is a command plus a place in the order, and the command runs on a machine Airflow controls while the data stays wherever the command points. That division is why an Airflow deployment can run 2,000 pipelines a night without holding a single row of business data.
- Airflow owns
- When each run begins
- Which task may start, given what has finished
- The record of every attempt and its log
- The picture of the whole, in one place
- The systems it calls own
- The bookings and reviews themselves
- The query that aggregates them
- The file the export writes
- The dashboard that shows the result
About this lesson
This section is part of the DAG Anatomy: Beginner lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.