The System Tour
Concepts covered: Pipeline Monitoring
When they ask you to walk through the system
A common first follow-up is 'can you describe the system around that job?' The interviewer is placing your fix: what feeds the job, how large each input is, where the output lands and who waits for it. A tour of about 30 seconds that names the sizes lets them see the problem before you explain it, which is the strongest place a war story can put a listener.
- Store tills (source, hourly order-line files)
- orders (storage, Parquet, 410 GB for 90 days)
- stores (storage, 1,200 rows, 3 MB)
- store_revenue_nightly (transform, Spark, 40 executors); freshness: done by 05:00; parallelism: 200 partitions
- store_revenue (storage, 540 rows per run)
- Regional dashboards (consumer, refresh at 05:00)
- Store tills -> orders
- orders -> store_revenue_nightly
- stores -> store_revenue_nightly
- store_revenue_nightly -> store_revenue
- store_revenue -> Regional dashboards
- Sources: Store tills
- Raw: orders, stores
- Processing: store_revenue_nightly
- Serving: store_revenue, Regional dashboards
Six boxes cover it. The tour leaves out the file layout, the scheduler and the monitoring, because none of them touch the join. Add a box only when the fix depends on it.
Sizes are what make the tour useful
A tour without sizes is a list of names. The fact that unlocks Priya's story is the gap between the 2 inputs to the join: 410 GB on one side, 3 MB on the other. An engineer who hears those 2 sizes next to the words 'sort-merge join' can predict the fix before Priya names it.
- orders: order_line_id BIGINT PK, order_date DATE, store_id INTEGER FK, product_id BIGINT, quantity INTEGER, amount DECIMAL(12,2)
- stores: store_id INTEGER PK, store_name VARCHAR, region VARCHAR, opened_on DATE
- orders.store_id references stores.store_id (N:1)
Say the join key and its direction as well: many order lines to a single store. That relationship is part of why the broadcast is safe. Each order line matches at most one store, so the join adds the region column without multiplying rows, and the joined data stays the size of orders.
When you do not remember a size
If you no longer have the sizes written down, recover them before the interview. The Stages tab shows bytes read in the Input column of each scan stage, and the scan node in the SQL tab reports its output rows and the size of files read. When you have only one of the 2, derive the other: about 100 million order lines a day for 90 days is 9 billion lines, and 410 GB spread over 9 billion lines is roughly 46 bytes per compressed line. A rounded figure such as 'over 400 GB' serves the tour well. A missing one leaves the interviewer unable to see why the join was the problem.
The tour, in the order data flows
- Source: Hourly files of order lines, about 100 million lines a day across 1,200 stores.
- Inputs: orders holds 410 GB for the 90 days the job reads; stores is a 3 MB dimension, a row per store.
- The job: Joins orders to stores on store_id and sums amount by region and day, on 40 executors with 200 shuffle partitions.
- Output: Six regions times 90 days, 540 rows, overwritten each night.
- Consumer: Refresh at 05:00, an hour after the job starts, which sets the deadline.
Upstream to downstream is the order to use, because it is the order the interviewer will draw in their head. Starting at the dashboards and working backwards forces them to hold the whole picture before any of it connects.
Finish on the box that shuffled
End the tour by pointing at where the data moved: 'The join inside the job is where it shuffled, and that is the box I changed.' That sentence ties the system to your action and hands the interviewer the question you want next: why that box?
- A tour that helps
- Names each input with its size
- Says the deadline once
- Marks where the shuffle happens
- Stops at the consumer
- A tour that stalls
- Lists tools with no sizes
- Explains the company's data strategy
- Covers every job in the pipeline
- Never says where the fix sits
About this lesson
This section is part of the The War Story lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.