Ingestion Patterns: Advanced

Dual writes, outbox, backpressure, and multi-tenant isolation decide whether ingestion survives scale

Dual writes, outbox, backpressure, and multi-tenant isolation decide whether ingestion survives scale

Category

Pipeline Architecture

Difficulty

advanced

Duration

38 minutes

Challenges

0 hands-on challenges

Topics covered: The Dual-Write Problem, The Transactional Outbox Pattern, Backpressure at the Ingestion Layer, Multi-Tenant Ingestion Fairness, Dual-Write to Outbox + CDC

Lesson Sections

1. The Dual-Write Problem (concepts: paDualWrite)

   Dual write is the most common production bug in event-driven architectures. An application writes to its database and to a message broker in the same handler, expecting both writes to succeed or both to fail. There is no transaction that spans the two systems. One will eventually succeed without the other. The bug is not in the code. The bug is in the architecture. The transactional guarantee the application thinks it has does not exist. What a Dual Write Looks Like The function looks correct on

2. The Transactional Outbox Pattern (concepts: paTransactionalOutbox, paIdempotency)

   The transactional outbox is the canonical fix for the dual-write problem. The application writes only to the database. Inside the same transaction that writes the business row, it inserts a row into an outbox table that describes the event to be published. A separate publisher process reads from the outbox table and publishes events to the broker. The atomic boundary is the database transaction; the broker is downstream of that boundary. Either both rows commit or neither does, and the publisher

3. Backpressure at the Ingestion Layer (concepts: paBackpressure)

   Backpressure is what happens when a source produces faster than the pipeline consumes. The mismatch is normal in steady state; it is the spike that exposes the design. Without explicit backpressure handling, the slowest component in the chain becomes a buffer, fills, and then either drops events, crashes, or amplifies the spike upstream. Designing for backpressure is the difference between a pipeline that absorbs a 10x burst and a pipeline that becomes the incident. Where the Mismatch Shows Up T

4. Multi-Tenant Ingestion Fairness (concepts: paMultiTenant)

   A platform that ingests data on behalf of many producers is multi-tenant. The producers might be customer accounts (Segment ingesting events for thousands of companies), internal teams (an internal data platform serving every product team), or external partners (a marketplace ingesting from every seller). The defining property is that one ingestion infrastructure serves N producers, and the producers do not coordinate with each other. Multi-tenant ingestion has problems single-tenant ingestion d

5. Dual-Write to Outbox + CDC (concepts: paDualWrite, paTransactionalOutbox, paCdc)

   The synthesis exercise is a real-shaped migration. A logistics company runs an order service backed by Postgres. The service publishes order_placed, order_shipped, and order_delivered events to a downstream warehouse, a recommendation system, and a customer notification service. Three years ago the team wired this up with the dual write inside each handler. The reconciliation job catches roughly 0.4% of events that drift, but enough leak through to drive customer complaints. The new principal en