Multi-Tenant Ingestion Fairness

A multi-tenant ingestion layer runs many customers' data through the same workers and queues and writes it to shared sinks. Sharing keeps hardware busy and cheap, and lets the largest tenant's worst day land on every tenant. Fairness means each tenant's freshness depends on its own load and plan, whatever its neighbors are doing.

One resync, 300 tenants waiting

A shared queue serves 301 tenants. At 09:00, tenant acme requests a full resync, and 50M rows enter the queue. The other 300 tenants each add 2,000 rows every 15 minutes, and the workers process 200,000 rows per minute.

SchedulerA small tenant's 09:05 batch waitsacme's resync finishes after
First in, first out250 minutes or moreAbout 250 minutes
Fair across tenants3 minutes at mostAbout 313 minutes

Fairness costs acme about an hour and saves each small tenant more than 4 hours. The small tenants' rows use 3 minutes of every 15, and acme gets the rest, because a fair scheduler is work-conserving: capacity a tenant leaves unused goes to whoever has work.

Deficit round robin across tenant queues

Fair scheduling needs a queue per tenant and a scheduler that hands out capacity in rounds. In deficit round robin, each tenant with work earns a quantum of rows per round, scaled by its weight, and spends that credit on its queued batches:

def schedule_round(queues, weights, deficit, quantum=2000):
    for tenant, queue in queues.items():
        if not queue:
            deficit[tenant] = 0
            continue
        earned = quantum * weights.get(tenant, 1)
        deficit[tenant] = deficit.get(tenant, 0) + earned
        while queue and queue[0].rows <= deficit[tenant]:
            batch = queue.popleft()
            deficit[tenant] -= batch.rows
            yield tenant, batch

Weights express plan tiers: a weight-4 tenant gets 4 times a weight-1 tenant's share while both have work. Resetting deficit for empty queues stops an idle tenant from banking credit and bursting later. A batch larger than its credit waits extra rounds, so large loads should arrive as keyset chunks.

When sharing itself is the risk

A scheduler divides worker time and nothing else: it cannot stop an oversized file from crashing a shared worker, or a single tenant from filling the buffer. Each stronger level of isolation costs more capacity:

LevelHow it worksProtects againstCost
Fair schedulingPer-tenant queues sharing a worker pool by weightQueue starvationLowest; a bad payload can still crash shared workers
Per-tenant quotasA token bucket per tenant on rows or bytes per second at admissionA single tenant flooding the bufferTenants over quota wait or get 429
BulkheadsSeparate worker pools per tier or for the largest tenantsCrashes and memory blowups spreadingIdle capacity in every pool
Dedicated cellA full ingestion stack for a single tenantEvery shared resourceHighest; for tenants whose load would dominate any pool

Per-tenant observability

Averages hide the tenant in trouble. With 300 tenants at 2 minutes of lag and a single tenant at 6 hours, mean lag reads around 3.2 minutes against a 15-minute SLA, and the dashboard stays green. Track freshness and dead-letter rate per tenant. Compare each tenant's volume against its own history and alert on its schema changes, then report the worst tenant and the count past SLA:

SELECT
    tenant_id,
    max(lag_seconds) AS worst_lag_seconds,
    sum(rows_ingested) AS rows_today,
    sum(rows_dead_lettered) AS rows_dead_lettered
FROM ingest_batches
WHERE batch_date = DATE '2026-10-24'
GROUP BY tenant_id
HAVING max(lag_seconds) > 900
ORDER BY worst_lag_seconds DESC

Carry tenant_id in every record's structure

Ingestion stamps tenant_id at admission and rejects records without it. From there it is a required envelope field, part of every table's primary key and a partition column in storage. Recovering the tenant later by joining to an accounts table fails when it matters most: when 2 tenants each have an order 1001, when an account moves between tenants, or when a departing tenant's rows must all be found and deleted.

Schema
  • orders: tenant_id VARCHAR PK, order_id BIGINT PK, status VARCHAR, created_at TIMESTAMP
  • order_items: tenant_id VARCHAR PK, order_id BIGINT PK, line_no INTEGER PK, sku VARCHAR, quantity INTEGER
  • order_items.tenant_id references orders.tenant_id (N:1)
  • order_items.order_id references orders.order_id (N:1)

Tenant boreal uploads 40 GB JSON files that exhaust worker memory. Each crash fails every batch on that worker, whichever tenants they belong to. Fair scheduling is already in place. What's the smallest change that protects the other tenants?

About this lesson

This section is part of the Ingestion Patterns: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.