Multi-Tenant Ingestion Fairness
A multi-tenant ingestion layer runs many customers' data through the same workers and queues and writes it to shared sinks. Sharing keeps hardware busy and cheap, and lets the largest tenant's worst day land on every tenant. Fairness means each tenant's freshness depends on its own load and plan, whatever its neighbors are doing.
One resync, 300 tenants waiting
A shared queue serves 301 tenants. At 09:00, tenant acme requests a full resync, and 50M rows enter the queue. The other 300 tenants each add 2,000 rows every 15 minutes, and the workers process 200,000 rows per minute.
| Scheduler | A small tenant's 09:05 batch waits | acme's resync finishes after |
|---|---|---|
| First in, first out | 250 minutes or more | About 250 minutes |
| Fair across tenants | 3 minutes at most | About 313 minutes |
Fairness costs acme about an hour and saves each small tenant more than 4 hours. The small tenants' rows use 3 minutes of every 15, and acme gets the rest, because a fair scheduler is work-conserving: capacity a tenant leaves unused goes to whoever has work.
Deficit round robin across tenant queues
Fair scheduling needs a queue per tenant and a scheduler that hands out capacity in rounds. In deficit round robin, each tenant with work earns a quantum of rows per round, scaled by its weight, and spends that credit on its queued batches:
def schedule_round(queues, weights, deficit, quantum=2000):
for tenant, queue in queues.items():
if not queue:
deficit[tenant] = 0
continue
earned = quantum * weights.get(tenant, 1)
deficit[tenant] = deficit.get(tenant, 0) + earned
while queue and queue[0].rows <= deficit[tenant]:
batch = queue.popleft()
deficit[tenant] -= batch.rows
yield tenant, batchWeights express plan tiers: a weight-4 tenant gets 4 times a weight-1 tenant's share while both have work. Resetting deficit for empty queues stops an idle tenant from banking credit and bursting later. A batch larger than its credit waits extra rounds, so large loads should arrive as keyset chunks.
When sharing itself is the risk
A scheduler divides worker time and nothing else: it cannot stop an oversized file from crashing a shared worker, or a single tenant from filling the buffer. Each stronger level of isolation costs more capacity:
| Level | How it works | Protects against | Cost |
|---|---|---|---|
| Fair scheduling | Per-tenant queues sharing a worker pool by weight | Queue starvation | Lowest; a bad payload can still crash shared workers |
| Per-tenant quotas | A token bucket per tenant on rows or bytes per second at admission | A single tenant flooding the buffer | Tenants over quota wait or get 429 |
| Bulkheads | Separate worker pools per tier or for the largest tenants | Crashes and memory blowups spreading | Idle capacity in every pool |
| Dedicated cell | A full ingestion stack for a single tenant | Every shared resource | Highest; for tenants whose load would dominate any pool |
Per-tenant observability
Averages hide the tenant in trouble. With 300 tenants at 2 minutes of lag and a single tenant at 6 hours, mean lag reads around 3.2 minutes against a 15-minute SLA, and the dashboard stays green. Track freshness and dead-letter rate per tenant. Compare each tenant's volume against its own history and alert on its schema changes, then report the worst tenant and the count past SLA:
SELECT
tenant_id,
max(lag_seconds) AS worst_lag_seconds,
sum(rows_ingested) AS rows_today,
sum(rows_dead_lettered) AS rows_dead_lettered
FROM ingest_batches
WHERE batch_date = DATE '2026-10-24'
GROUP BY tenant_id
HAVING max(lag_seconds) > 900
ORDER BY worst_lag_seconds DESCCarry tenant_id in every record's structure
Ingestion stamps tenant_id at admission and rejects records without it. From there it is a required envelope field, part of every table's primary key and a partition column in storage. Recovering the tenant later by joining to an accounts table fails when it matters most: when 2 tenants each have an order 1001, when an account moves between tenants, or when a departing tenant's rows must all be found and deleted.
- orders: tenant_id VARCHAR PK, order_id BIGINT PK, status VARCHAR, created_at TIMESTAMP
- order_items: tenant_id VARCHAR PK, order_id BIGINT PK, line_no INTEGER PK, sku VARCHAR, quantity INTEGER
- order_items.tenant_id references orders.tenant_id (N:1)
- order_items.order_id references orders.order_id (N:1)
Tenant boreal uploads 40 GB JSON files that exhaust worker memory. Each crash fails every batch on that worker, whichever tenants they belong to. Fair scheduling is already in place. What's the smallest change that protects the other tenants?
- Move boreal's files to a separate worker pool sized for them, so its crashes stay inside its own bulkhead.
- Raise boreal's scheduling weight so its files finish sooner and spend less time on shared workers.
- Give boreal a dedicated ingestion cell with its own queues, workers, sinks and buffer.
- Add a per-tenant quota on boreal's rows per second at admission.
About this lesson
This section is part of the Ingestion Patterns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.