The API Drip Feed

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Seniority

L5

Problem

We pull tasks, users, and projects from a third-party project management SaaS API into our analytics warehouse, but the API only serves small rate-limited pages and bursts unevenly, so ingestion has to absorb those bursts without dropping records. The rate limit won't cover pulling every table fresh each hour, so only the one tracked dashboard's task updates can ride a within-the-hour path while the rest of the tables stay on a once-a-day refresh. The vendor reshapes the API's fields every few weeks, so schema changes have to be caught before they reach the warehouse and surfaced to the team instead of silently corrupting the tables.

Summary

The API gives you 100 records at a time. You need millions.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.