API Patterns

The advanced probe hands you a vendor API and a figure: 'We need all 10 million records, the API allows 1,000 requests per hour and returns 100 records per page. Design the ingestion.' Candidates who start describing a loop have missed the question. The interviewer wants the arithmetic first, because the arithmetic rules out the design most people begin with.

• 100,000
• 100 h

Once a complete copy takes longer than 4 days, the design has to avoid complete copies. Only 2 directions survive: an incremental filter such as updated_since, which shrinks each run to the records that changed, and a bulk export endpoint if the vendor offers one, which trades API calls for a file drop. The incremental path carries every hazard from the batch section, including clock drift between the vendor's servers and the need for an overlap window, and saying so is the connection the interviewer hopes to hear.

Offset versus cursor pagination

Offset pagination fails in a way tutorials rarely show. Suppose the feed is sorted newest first and page 37 asks for positions 3,600 through 3,699. If the vendor creates 2 records before you request page 38, every existing record moves 2 positions later: the old 3,698 and 3,699 become 3,700 and 3,701 and arrive again at the top of page 38. Deletes do the reverse. If 2 records you already read are deleted, the first 2 records of page 38 slide back onto page 37's range and are never read. Inserts produce repeats and deletes produce gaps. Sorted oldest first, new records land at the end and shift nothing, but a feed sorted by update time is the worst case: an edited record you already read leaves its old position, which shifts the rest back and skips 1, then reappears at the end, where you read it again. A cursor hands back an opaque token marking a position in the data rather than a count, so neither inserts nor deletes can move it.

import random
import time


def fetch_pages(session, url, cursor=None):
    delay = 1.0
    while True:
        params = {"limit": 100, "updated_since": "2026-10-18T02:00:00Z"}
        if cursor:
            params["cursor"] = cursor
        resp = session.get(url, params=params, timeout=30)
        if resp.status_code == 429:
            wait = float(resp.headers.get("Retry-After", delay))
            time.sleep(wait + random.uniform(0, 1))
            delay = min(delay * 2, 60.0)
            continue
        resp.raise_for_status()
        delay = 1.0
        page = resp.json()
        cursor = page.get("next_cursor")
        yield page["items"], cursor
        if cursor is None:
            return

The interviewer scores 2 details in that loop. The generator yields the cursor alongside each page so the caller can persist it after the page is written to raw storage; a crash on page 612 then resumes at 612 instead of page 1 with 611 wasted requests. And the retry honours the Retry-After header before falling back to doubling with jitter, because clients that retry on a fixed schedule synchronise with each other and keep the vendor's limiter saturated for everyone.

Webhooks and the reconciliation poll

When the interviewer offers webhooks as the alternative, the scored answer treats them as the fast path and keeps a slow path beside them. A webhook arriving during your outage is retried by the vendor for a bounded period and then dropped, and vendors do send the same event twice. The pattern is to accept the webhook, write the raw body, acknowledge immediately, and run a reconciliation poll on a schedule with the change filter set a day back, deduplicating on the vendor's event id. The poll is the batch leg of a small hybrid, which is where the next section begins.

• 'What if the vendor changes a field type?' Land the raw response first and parse in a separate step you can rerun.
• 'How do you know you received everything?' Compare against a count endpoint, or run the reconciliation poll.
• 'The rate limit is shared with 3 other teams.' One central client holds the budget; 4 independent ones fight over it.
• 'Which time zone is updated_since in?' Send UTC, record what the vendor returns, and keep the overlap window.

About this lesson

This section is part of the How Data Moves: Intermediate lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.