'How does your pipeline handle bad data?' is asked in every system design round, and most candidates fumble it. They either say 'validate the data' (too vague) or 'reject bad records' (too destructive). The correct answer is a dead letter queue: a sidecar destination that captures failed records with their error metadata so the pipeline continues processing good data while bad data is isolated, diagnosed, and replayed after the root cause is fixed. Without a DLQ, a single corrupted message can crash a consumer repeatedly, stalling all downstream processing.
What you will be able to do
Recognize DLQ as the standard answer to every 'bad data' question
Design the retry-then-DLQ flow with specific error classification
Frame DLQ volume as a data quality signal, not just an error bin
"How Does Your Pipeline Handle Bad Data?"
Daily Life
Interviews
You are being tested on DLQ when you hear:
▸"What happens when you receive malformed data?"
▸"A record fails to parse. What do you do?"
▸"How do you handle poison pill messages?"
▸"Your pipeline is stuck on one bad record. How do you unblock it?"
▸"How do you ensure bad data doesn't corrupt good data?"
▸Any question about error handling, data quality, or pipeline resilience
What They're Really Testing
The hidden rubric: does this candidate design pipelines that continue processing good data when bad data appears, or does one bad record stop everything? The 'resilience' section of the FAANG scorecard specifically evaluates how you handle the unhappy path. Candidates who only present the happy path are the most common failure mode cited by interviewers.
The Unlock
A DLQ is not an error log. It is a parallel processing path. Good records flow through the main pipeline. Bad records are diverted to the DLQ with the full error context (original record, error message, stack trace, timestamp, retry count). The DLQ is a queue, not a graveyard. Records in it are expected to be replayed after the root cause is fixed.
The 60-Second Framework
Classify the error: 'Is it transient (network timeout, temporary unavailability) or permanent (schema mismatch, null required field)?'
Transient: retry with exponential backoff (1s, 2s, 4s, 8s) up to a max retry count.
Permanent or max retries exceeded: divert to the DLQ with full error metadata.
Alert: 'DLQ volume > threshold triggers a PagerDuty alert. The rate of DLQ writes is a data quality signal.'
Replay: 'After fixing the root cause, replay DLQ records through the main pipeline with idempotent writes.'
This five-step flow takes 60 seconds to articulate and hits every rubric item: error classification, retry strategy, isolation, monitoring, and recovery. Most candidates stop at step 3. Steps 4 and 5 are the strong-hire signals.
Why Companies Care
At Uber, a malformed ride event caused a Kafka consumer to crash-loop for 4 hours, blocking 200,000 subsequent events from processing. A DLQ would have isolated the one bad record and let the other 200,000 flow. At Stripe, every event processing pipeline has a mandatory DLQ. The DLQ error rate is reviewed in weekly data quality standups. At Netflix, DLQ volume by error category is displayed on the team's Grafana dashboard and feeds into upstream schema contract enforcement.
DLQ Architecture: Divert, Store, Alert
Daily Life
Interviews
A DLQ is a separate storage destination (Kafka topic, SQS queue, S3 bucket) that captures failed records alongside their error metadata. The design has three components: diversion logic, storage schema, and alerting.
Kafka
events
Quality
parse validate
Spark
main pipeline
DLQ topic
dlq
Snowflake
warehouse
PagerDuty
alert
Good records flow through the main pipeline; the validation gate diverts bad records to the DLQ with full error context, then alerts. One poison-pill record never blocks the other 200,000.
DLQ Record Schema
1
{
2
"original_record":{
3
"user_id":"abc",
4
"event":"click",
5
"ts":"not-a-date"
6
},
7
"error":{
8
"type":"ParseError",
9
"message":"Cannot parse 'not-a-date' as ISO timestamp",
10
"stack_trace":"...",
11
"pipeline":"clickstream_etl",
12
"stage":"parse_timestamp"
13
},
14
"metadata":{
15
"source_topic":"raw_clicks",
16
"source_partition":7,
17
"source_offset":1482903,
18
"retry_count":3,
19
"first_failure_at":"2024-03-15T10:23:45Z",
20
"last_failure_at":"2024-03-15T10:24:12Z"
21
}
22
}
The raw input that failed. Without it, you cannot replay. Storing only the error message without the record is a common anti-pattern.
Which processing step failed. A parse error in stage 1 has a different root cause than a validation error in stage 3.
The exact position in the source. Enables targeted replay of specific records rather than replaying the entire topic.
How many times this record was retried before DLQ diversion. If retry_count is always 1, your transient error detection is too aggressive.
Storage Options
Storage
Best For
Queryability
Retention
Kafka topic (dlq_*)
Streaming pipelines, easy replay to main topic
Low (need consumer to read)
Configurable retention
S3 + Athena
Long-term analysis, large DLQ volumes
High (SQL via Athena/Presto)
Indefinite, cheap storage
SQS DLQ
AWS-native, built-in retry/DLQ support
Medium (SQS console)
14-day max retention
Database table
Small volumes, operational dashboards
High (direct SQL)
Must manage table growth
•No Hire
"Log the error and skip the record"
No mention of storing the original record
No alerting on DLQ volume
✓Strong Hire
"Divert to a DLQ topic with the original record, error metadata, and source offset"
"Alert on DLQ write rate > 1% of total throughput"
"DLQ records are replayable. After fixing root cause, push them back through the main pipeline."
Retry Strategies and Poison Pills
Daily Life
Interviews
Not all errors are the same. Transient errors (network timeout, temporary database lock) should be retried. Permanent errors (malformed schema, null in a NOT NULL field) will never succeed no matter how many times you retry. The interview tests whether you can classify errors and route them differently.
Retry Queue vs Dead Letter Queue
•Retry Queue
For transient errors: timeouts, throttling, temporary unavailability
Retry with exponential backoff: 1s, 2s, 4s, 8s
Add jitter to prevent thundering herd
Cap at 3-5 retries, then promote to DLQ
Records are expected to eventually succeed
•Dead Letter Queue
For permanent errors: schema mismatch, business rule violation, corrupted payload
No automatic retry. Requires human investigation.
Records are stored with full error context
Replayed manually after root cause fix
Records may never be reprocessable if the source is wrong
Poison Pill Detection
A poison pill is a message that causes the consumer to crash every time it tries to process it. Without detection, the consumer enters a crash loop: read poison pill, crash, restart, read same poison pill, crash again. The pipeline is stuck.
Pattern 1Pattern 2Pattern 3
Pattern 1
Max retry count per record
Track retry count per message (via header or external store). After N failures, divert to DLQ regardless of error type. This catches unexpected permanent errors that don't match your classification logic.
Pattern 2
Error type classification
Wrap processing in try/catch. Classify the exception: ParseException = permanent (DLQ immediately). TimeoutException = transient (retry). Unknown = retry up to N, then DLQ.
Pattern 3
Circuit breaker on error rate
If > 50% of records in a batch fail, something systemic is wrong (upstream schema change, dependency outage). Stop processing entirely, alert, and wait for investigation rather than filling the DLQ.
TIP
The senior move: 'I would distinguish three error categories: transient (retry), permanent (DLQ immediately), and systemic (circuit breaker). A systemic error like a schema change affecting 100% of records should not be handled record-by-record. It needs a pipeline-level pause and an alert to the upstream team.'
DLQ Monitoring and Reprocessing
Daily Life
Interviews
A DLQ without monitoring is a data graveyard. Records enter and nobody notices. The DLQ becomes a slowly growing pile of lost data that surfaces months later when a VP asks 'why are our numbers 3% lower than the source system?' The monitoring and reprocessing workflow is what makes a DLQ operational, not just architectural.
DLQ Monitoring Dashboard
Metric
Alert Threshold
Meaning
DLQ write rate
> 1% of total throughput
Bad data rate exceeds acceptable level
DLQ depth (unprocessed)
> 10,000 records
Records are accumulating without investigation
DLQ record age
Oldest record > 7 days
Records are being ignored
Error type distribution
New error type appears
New failure mode; possible upstream change
DLQ write rate spike
5x increase in 15 min
Systemic issue; possible schema break
Reprocessing Workflow
Investigate: Query DLQ records by error type. Identify root cause (schema change, bug, bad data from source).
Fix: Deploy the fix to the main pipeline (new parser, updated schema, validation logic).
Replay: Push DLQ records back through the main pipeline. The pipeline's idempotent writes ensure no duplicates.
Verify: Compare record counts before and after replay. DLQ depth should decrease to zero for that error type.
Postmortem: Update the error classification logic if the error was mis-categorized. Update upstream contracts if the error originated from a producer.
Step 5 is the L6 signal. Connecting DLQ analysis back to upstream contracts shows you think about the system holistically, not just your own pipeline. 'The DLQ told us the payments team changed their timestamp format. I updated our parser AND opened a ticket with them to add this field to their schema contract so we get advance notice next time.'
The Follow-Up Trap
Follow-UpFollow-Up
Follow-Up
"What if DLQ replay creates duplicates?"
Strong answer: 'The main pipeline uses idempotent writes (MERGE by event_id). Replaying a DLQ record that already succeeded in the main path will match on the key and make zero changes. Idempotency makes replay safe.'
Follow-Up
"What if the DLQ itself fills up?"
Strong answer: 'I'd tier the DLQ. Hot DLQ (Kafka topic, 7-day retention) for recent failures. Cold DLQ (S3) for archival. If the hot DLQ exceeds depth threshold, alert and circuit-break the pipeline until investigation.'
DLQ as a Data Quality Signal
Daily Life
Interviews
The senior insight that most candidates miss: the DLQ is not just an error handler. It is a data quality feedback loop. DLQ error categories and volume trends tell you which upstream producers are degrading, which schema contracts are being violated, and where your pipeline's assumptions no longer hold.
The Bridge Move
Bridge to Data ContractsBridge to ObservabilityBridge to SLA
Bridge to Data Contracts
"DLQ errors inform upstream schema contracts"
If 80% of DLQ records fail on a timestamp format change, that field needs a contract: 'timestamp is ISO 8601, UTC, non-null.' The DLQ is the evidence that justifies the contract.
Bridge to Observability
"DLQ rate is a leading indicator"
DLQ write rate increases before downstream metrics degrade. A 5% DLQ rate today means 5% missing data in tomorrow's reports. The DLQ is the canary in the data mine.
Bridge to SLA
"DLQ age is an SLA metric"
If the SLA says 'all data processed within 24 hours,' DLQ records older than 24 hours are SLA violations. DLQ age is not just a monitoring metric; it is a contractual obligation.
Vocabulary That Signals Seniority
Junior Phrasing
Senior Phrasing
"Log the error and skip it"
"Divert to a DLQ with the original record, error metadata, and source offset for replay"
"We'd retry a few times"
"Exponential backoff with jitter for transient errors, immediate DLQ for permanent errors, circuit breaker for systemic failures"
"We'd fix the bug and reprocess"
"Replay DLQ records through the main pipeline with idempotent writes, then verify counts match"
"The DLQ catches errors"
"The DLQ is a data quality feedback loop. Error categories inform upstream contract enforcement and producer-side fixes."
"We'd monitor the DLQ"
"DLQ write rate, depth, age, and error type distribution on a Grafana dashboard with PagerDuty alerts on threshold breaches"
Red Flag Phrases
"Just log the error and skip it" - Logging without storing the original record means you cannot replay. The data is lost forever.
"We'd stop the pipeline on bad data" - Stopping the entire pipeline for one bad record blocks all good data. A DLQ isolates failures.
"The DLQ will handle it" - A DLQ without monitoring, alerting, and a replay workflow is just a data graveyard. Records enter and nobody notices.
"We'd fix the bug and the data will self-correct" - Historical bad records in the DLQ do not self-correct. They must be explicitly replayed.
The two sentences that close every DLQ answer:
▸"The DLQ write rate is a data quality KPI reviewed in weekly standups."
▸"Every DLQ error category maps to an upstream contract violation that we report back to the producer team."
❯❯❯PUTTING IT ALL TOGETHER
> You are in an Uber data engineering interview. The interviewer asks: 'A malformed ride event crashes your consumer. How do you handle it?'
You say: 'I'd wrap processing in a try/catch that classifies the error. A parse error is permanent: divert to the DLQ immediately with the original record and error context. The consumer advances past the bad record and continues processing.'
The interviewer asks about 1,000 bad records. You say: 'If the DLQ write rate exceeds 1% of throughput, a circuit breaker pauses the pipeline and alerts the team. A 1,000-record burst suggests a systemic issue like an upstream schema change, not individual bad records.'
You bridge: 'After fixing the root cause, I replay DLQ records through the main pipeline. Idempotent writes ensure no duplicates. I then update the upstream contract to require advance notice of schema changes.'
KEY TAKEAWAYS
DLQ is the standard answer: every pipeline must handle bad data; DLQ isolates failures without blocking good data
Three error classes: transient (retry), permanent (DLQ), systemic (circuit breaker)
DLQ records are replayable: store original record + error metadata + source offset for targeted replay
Monitor DLQ health: write rate, depth, age, error type distribution
DLQ as feedback loop: error categories inform upstream schema contracts and producer fixes
Bad records kill pipelines; DLQs let you isolate failures without stopping the world
Category
Pipeline Architecture
Difficulty
advanced
Duration
25 minutes
Challenges
0 hands-on challenges
Topics covered: "How Does Your Pipeline Handle Bad Data?", DLQ Architecture: Divert, Store, Alert, Retry Strategies and Poison Pills, DLQ Monitoring and Reprocessing, DLQ as a Data Quality Signal
What They're Really Testing The Unlock A DLQ is not an error log. It is a parallel processing path. Good records flow through the main pipeline. Bad records are diverted to the DLQ with the full error context (original record, error message, stack trace, timestamp, retry count). The DLQ is a queue, not a graveyard. Records in it are expected to be replayed after the root cause is fixed. The 60-Second Framework This five-step flow takes 60 seconds to articulate and hits every rubric item: error c
A DLQ is a separate storage destination (Kafka topic, SQS queue, S3 bucket) that captures failed records alongside their error metadata. The design has three components: diversion logic, storage schema, and alerting. DLQ Record Schema Storage Options
Not all errors are the same. Transient errors (network timeout, temporary database lock) should be retried. Permanent errors (malformed schema, null in a NOT NULL field) will never succeed no matter how many times you retry. The interview tests whether you can classify errors and route them differently. Retry Queue vs Dead Letter Queue Poison Pill Detection A poison pill is a message that causes the consumer to crash every time it tries to process it. Without detection, the consumer enters a cra
A DLQ without monitoring is a data graveyard. Records enter and nobody notices. The DLQ becomes a slowly growing pile of lost data that surfaces months later when a VP asks 'why are our numbers 3% lower than the source system?' The monitoring and reprocessing workflow is what makes a DLQ operational, not just architectural. DLQ Monitoring Dashboard Reprocessing Workflow Step 5 is the L6 signal. Connecting DLQ analysis back to upstream contracts shows you think about the system holistically, not
The senior insight that most candidates miss: the DLQ is not just an error handler. It is a data quality feedback loop. DLQ error categories and volume trends tell you which upstream producers are degrading, which schema contracts are being violated, and where your pipeline's assumptions no longer hold. The Bridge Move Vocabulary That Signals Seniority Red Flag Phrases
