Latency vs Throughput Tradeoff

Latency is the time between an event happening and its result reaching the consumer. Throughput is how much data the pipeline processes per unit of time, measured in rows per second or gigabytes per hour. A pipeline can have high throughput and terrible latency: a nightly job that chews through 180GB in 50 minutes moves data fast, yet an order placed at 00:05 waits almost 24 hours before a dashboard shows it.

Keep latency and throughput apart by asking what each one answers. Latency answers how old is the newest row the consumer can see. Throughput answers how many rows per second can this pipeline absorb before falling behind. Neither number tells you the other.

All 3 pipelines carry the same 4M rows a day. The nightly load has the highest throughput, because it processes all 4M in under a minute of wall clock, and the worst latency. The click stream has the lowest throughput by a factor of 1,600 and the best latency.

Latency and throughput get confused because shortening a schedule lowers latency, and people read that as the pipeline getting faster. The processing rate did not change. Running the nightly load every hour makes each run handle 1/24 of the rows in roughly 1/24 of the time, so throughput during the run is unchanged while worst-case latency drops from 24 hours to 1.

Which one is holding you back

A pipeline is latency-constrained when it keeps up with the input but the consumer still gets results later than it needs them. The fix is a shorter schedule or a different movement pattern, since the machines sit idle most of the day. A pipeline is throughput-constrained when input arrives faster than the pipeline processes it, so the backlog grows. The fix is more parallel workers or less work per row. A shorter schedule leaves rows arriving per hour and rows processed per hour unchanged, so it does not help, and each extra run adds its own startup and commit overhead.

The same pipeline can switch categories: an hourly job that processes 10GB today is latency-constrained; when the input triples it becomes throughput-constrained with no code change.

You can measure both from the curated table itself if every row carries 2 timestamps: when the event happened and when the pipeline wrote it. The gap between them, taken at the 95th percentile, is latency as the consumer experiences it. Rows written per hour of run time is throughput.

SELECT
  date_trunc('hour', loaded_ts) AS load_hour,
  count(*) AS rows_loaded,
  approx_percentile(date_diff('second', event_ts, loaded_ts), 0.95) AS p95_latency_s
FROM curated.orders
WHERE loaded_ts >= localtimestamp - INTERVAL '1' DAY
GROUP BY 1
ORDER BY 1

Read the result as a pair of signals. If rows_loaded spikes in a single hour and p95_latency_s sits near 86,000 in that same hour, you are looking at a nightly batch and the consumer is seeing day-old rows. If rows_loaded is smooth across all 24 hours and p95_latency_s is climbing hour over hour, rows are arriving faster than they are written and the backlog is growing.

A pipeline reads a queue every hour and writes to a curated table. Each run takes 48 minutes, up from 30 minutes 3 months ago, and the queue depth at the start of each run has been rising for weeks. The consumer wants rows within 2 hours. Which metric constrains this pipeline, and what change helps?

• Latency; switch to a 15 minute schedule so rows arrive sooner.
• Throughput; add parallel workers or cut work per row so each run finishes well inside the interval.
• Latency; the consumer's 2 hour need is already met, so no change is required.
• Throughput; lengthen the schedule to every 2 hours so each run has more time.

About this lesson

This section is part of the Batch vs Streaming: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.