# The Quiet Drift

> Two versions of the same truth.

Canonical URL: <https://datadriven.io/problems/the_quiet_drift>

Domain: Python · Difficulty: medium · Seniority: L4

## Problem

Two nightly imports of the same ledger should hold identical records keyed by `id_field`, but the upstream systems drift and rows diverge. Reconcile the two lists into four id-ordered buckets: ids only in `source_a`, ids only in `source_b`, ids in both with identical records, and ids in both whose records differ, reporting for that last bucket which non-id fields differ and each source's value, treating a field absent from one source as `None` there.

## Worked solution and explanation

### What this problem really is

Strip the ledger costume and this is a membership classification wearing a field-level diff on its back. Everyone gets the easy half: index both sources by id, then set-difference and intersection to split ids into `only_a`, `only_b`, and both. The half that separates candidates is the diff on the 'both' bucket. The trap is quietly diffing only the keys of the `source_a` record. Do that and a field that lives only in `source_b` never gets looked at, so a record that genuinely drifted gets filed under 'matches'. Get it wrong and a real data-quality problem ships clean, with no alert.

---

### Break down the requirements

#### Step 1: Index both sources by id

Build `map_a` and `map_b`: dicts keyed by `record[id_field]` so id lookups are O(1). The output has four keys: `only_a`, `only_b`, `matches`, `mismatches`.

#### Step 2: Membership classification with set operations

Use Python set operations on the two id sets: `only_a = ids_a - ids_b`, `only_b = ids_b - ids_a`, `both = ids_a & ids_b`. Sort the first two before returning.

#### Step 3: Field-level diff for matched ids

For each id in `both`, compare the two records field by field over the UNION of their non-id keys. If every non-id field is equal, the id goes in `matches`. Otherwise, build `{field: {'a': a_val, 'b': b_val}}` for each differing field and append `{'id': id, 'differences': ...}` to `mismatches`. The union is what saves you from the one-directional-diff trap: in the visible example id 2 carries `note` only in `source_b`, so it surfaces as `{'a': None, 'b': 'adjusted'}`.

#### Step 4: Sort and assemble

Return the four keys with `only_a`, `only_b`, `matches` sorted by id, and `mismatches` sorted by `id`. Iterating the intersection in sorted order gives you sorted `matches` and `mismatches` for free.

---

### The solution

**Hash-by-id, set diff for membership, field-by-field diff for content**

```python
def reconcile(source_a: list[dict], source_b: list[dict], id_field: str) -> dict:
    map_a = {r[id_field]: r for r in source_a}
    map_b = {r[id_field]: r for r in source_b}
    ids_a = set(map_a)
    ids_b = set(map_b)
    only_a = sorted(ids_a - ids_b)
    only_b = sorted(ids_b - ids_a)
    matches = []
    mismatches = []
    for rid in sorted(ids_a & ids_b):
        a, b = map_a[rid], map_b[rid]
        fields = (set(a) | set(b)) - {id_field}
        diffs = {}
        for f in fields:
            if a.get(f) != b.get(f):
                diffs[f] = {'a': a.get(f), 'b': b.get(f)}
        if diffs:
            mismatches.append({'id': rid, 'differences': diffs})
        else:
            matches.append(rid)
    return {
        'only_a': only_a,
        'only_b': only_b,
        'matches': matches,
        'mismatches': mismatches,
    }
```

> **Time and Space Complexity**
>
> **Time:** O(n + m + k * f) where n = |`source_a`|, m = |`source_b`|, k = matched ids, f = avg fields per record.
> 
> Space: O(n + m) for the two id-indexed maps.

> **Interviewers Watch For**
>
> Strong candidates diff over the **union** of field names per matched pair, so a field that exists in only one source still surfaces. They also keep `id` out of the diff loop so the match key is never reported as a difference.

> **Common Pitfall**
>
> Iterating only over the keys of the `source_a` record when diffing. If `source_b` carries a field `source_a` lacks, that drift is silently missed and the id is filed as a match. Using `set(a) | set(b)` per matched pair, and `dict.get` so a missing key reads as `None`, catches both directions.

---

## Common follow-up questions

- How would you present the diff in a human-readable report? _(Tests formatting diffs with old/new values in a table layout.)_
- What if records had nested structures? _(Tests recursive field comparison.)_
- How would this scale to millions of records? _(Tests streaming reconciliation with sorted merge or database-side comparison.)_
- What if the ID field were not unique in one of the lists? _(Tests handling multi-valued mappings or flagging the anomaly.)_

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/the_quiet_drift)
- [Python Interview Questions](https://datadriven.io/python-interview-questions)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.