The Quiet Drift

A medium Python interview practice problem on DataDriven. Write the Python and run it against test cases, with instant feedback.

Domain
Python
Difficulty
medium
Seniority
L4
Asked in
Amazon

Problem

Two nightly imports of the same ledger should hold identical records keyed by id_field, but the upstream systems drift and rows diverge. Reconcile the two lists into four id-ordered buckets: ids only in source_a, ids only in source_b, ids in both with identical records, and ids in both whose records differ, reporting for that last bucket which non-id fields differ and each source's value, treating a field absent from one source as None there.

Two versions of the same truth.

Example

Input:

source_a = [{"id":1,"amount":100,"status":"posted"},{"id":2,"amount":200,"status":"posted"},{"id":3,"amount":300,"status":"pending"}], source_b = [{"id":1,"amount":100,"status":"posted"},{"id":2,"note":"adjusted","amount":205,"status":"posted"},{"id":4,"amount":400,"status":"posted"}], id_field = "id"

Output:

{"only_a":[3],"only_b":[4],"matches":[1],"mismatches":[{"id":2,"differences":{"note":{"a":null,"b":"adjusted"},"amount":{"a":200,"b":205}}}]}

Worked solution and explanation

What this problem really is

Strip the ledger costume and this is a membership classification wearing a field-level diff on its back. Everyone gets the easy half: index both sources by id, then set-difference and intersection to split ids into only_a, only_b, and both. The half that separates candidates is the diff on the 'both' bucket. The trap is quietly diffing only the keys of the source_a record. Do that and a field that lives only in source_b never gets looked at, so a record that genuinely drifted gets filed under 'matches'. Get it wrong and a real data-quality problem ships clean, with no alert.

Break down the requirements

Step 1: Index both sources by id

Build map_a and map_b: dicts keyed by record[id_field] so id lookups are O(1). The output has four keys: only_a, only_b, matches, mismatches.

Step 2: Membership classification with set operations

Use Python set operations on the two id sets: only_a = ids_a - ids_b, only_b = ids_b - ids_a, both = ids_a & ids_b. Sort the first two before returning.

Step 3: Field-level diff for matched ids

For each id in both, compare the two records field by field over the UNION of their non-id keys. If every non-id field is equal, the id goes in matches. Otherwise, build {field: {'a': a_val, 'b': b_val}} for each differing field and append {'id': id, 'differences': ...} to mismatches. The union is what saves you from the one-directional-diff trap: in the visible example id 2 carries note only in source_b, so it surfaces as {'a': None, 'b': 'adjusted'}.

Step 4: Sort and assemble

Return the four keys with only_a, only_b, matches sorted by id, and mismatches sorted by id. Iterating the intersection in sorted order gives you sorted matches and mismatches for free.

The solution

Hash-by-id, set diff for membership, field-by-field diff for content
def reconcile(source_a: list[dict], source_b: list[dict], id_field: str) -> dict:
    map_a = {r[id_field]: r for r in source_a}
    map_b = {r[id_field]: r for r in source_b}
    ids_a = set(map_a)
    ids_b = set(map_b)
    only_a = sorted(ids_a - ids_b)
    only_b = sorted(ids_b - ids_a)
    matches = []
    mismatches = []
    for rid in sorted(ids_a & ids_b):
        a, b = map_a[rid], map_b[rid]
        fields = (set(a) | set(b)) - {id_field}
        diffs = {}
        for f in fields:
            if a.get(f) != b.get(f):
                diffs[f] = {'a': a.get(f), 'b': b.get(f)}
        if diffs:
            mismatches.append({'id': rid, 'differences': diffs})
        else:
            matches.append(rid)
    return {
        'only_a': only_a,
        'only_b': only_b,
        'matches': matches,
        'mismatches': mismatches,
    }

Common follow-up questions