# The Fuller Copy > The same listing lands twice, half-filled each time. Keep what is whole. Canonical URL: Domain: Python · Difficulty: medium · Seniority: mid ## Problem A scraping pipeline pulls product listings from several sites, and each raw record differs by source: a price may sit under the first entry of `offers` or as a top-level field, and it arrives as a string like `"$1,299.00"`, while the id and title hide behind different key names. Flatten each raw record into a uniform listing, coercing prices to floats and timestamps to a plain date, and leaving any field that is absent as `None`. The same listing can be scraped from more than one page under the same id, so when an id repeats, keep the copy with the fewest missing fields. ## Worked solution and explanation ### Why this problem exists in real interviews This is defensive extraction from heterogeneous records with a merge-on-conflict, dressed up as scraper normalization. The skill being probed: can you pull the same logical field out of records where it lives at a different path and under a different key from every source, coerce its type, and then collapse duplicates by choosing the copy that carries the most data? Anyone can write a dict comprehension. The two traps are (1) `record['offers'][0]['price']` blows up with KeyError or IndexError the instant a source omits `offers`, and (2) a dict keyed by id that overwrites on every hit keeps whichever copy was scraped last, so it silently throws away the price and date the other copy had. --- ### Break down the requirements #### Step 1: Extract every field through a fallback chain The id is under `id` on one site and `sku` on another; the title is `title` or `name`. Write one helper that walks a list of candidate keys and returns the first non-null value, so adding a new source's key name is a one-line change, not a rewrite. #### Step 2: Reach into offers defensively, then fall back The price prefers `offers[0].price` and falls back to a top-level `price`. `offers` is sometimes absent and sometimes an empty list, so guard with `isinstance(offers, list) and offers` before indexing `[0]`. Never index a path you have not proven exists. #### Step 3: Coerce types, treat missing as None Prices arrive as `"$1,299.00"`, `"1,050"`, or the int `1299`. Strip the currency symbol and thousands separators, then float it; pass ints straight through `float()`. A timestamp like `2026-07-01T12:00:00Z` becomes the date portion before the `T`. Anything absent stays `None` rather than crashing. #### Step 4: Deduplicate by keeping the fullest copy Group by the canonical id. When an id repeats, keep the flat record with the most non-null fields. Replace the stored copy only on a strict improvement, which keeps the first-seen copy on a tie and preserves first-appearance ordering for the output. --- ### The solution **Defensive flatten plus completeness-based dedup** ```python def normalize_listings(records): def first_present(rec, keys): for k in keys: val = rec.get(k) if val is not None: return val return None def parse_price(raw): if raw is None: return None if isinstance(raw, (int, float)): return float(raw) cleaned = raw.replace('$', '').replace(',', '').strip() return float(cleaned) if cleaned else None def parse_date(raw): return raw.split('T')[0] if raw else None def flatten(rec): offers = rec.get('offers') offer = offers[0] if isinstance(offers, list) and offers else {} price_raw = offer.get('price') if price_raw is None: price_raw = rec.get('price') return { 'id': first_present(rec, ['id', 'sku']), 'title': first_present(rec, ['title', 'name']), 'price': parse_price(price_raw), 'currency': offer.get('priceCurrency'), 'posted_at': parse_date(first_present(rec, ['datePosted', 'posted_at'])), 'source_site': rec.get('source_site'), } def completeness(flat): return sum(1 for v in flat.values() if v is not None) best = {} order = [] for rec in records: flat = flatten(rec) cid = flat['id'] if cid not in best: best[cid] = flat order.append(cid) elif completeness(flat) > completeness(best[cid]): best[cid] = flat return [best[cid] for cid in order] ``` > **Complexity** > > One pass over the records, O(n) in the number of records, with each flatten doing a constant number of dict lookups and a completeness scan over the fixed six-field schema (also constant). Space is O(u) for the u unique ids. Scraper batches are thousands to low millions of records per crawl, and this stays a single linear pass with no sorting, which is exactly what you want on an ingest step that runs on every page fetch. > **Interviewers Watch For** > > Whether you guard `offers` with an isinstance check before indexing (the field is sometimes a list, sometimes absent, occasionally an empty list), whether your key fallbacks make a new source a one-line addition instead of a rewrite, and whether your dedup rule is defensible. Strong candidates say out loud that they keep the most complete copy and replace only on a strict improvement so ties keep the first-seen record deterministically. > **Common Pitfall** > > Last-write-wins dedup: keying a dict by id and assigning on every record. It runs clean and looks right on a happy-path test, but when the second copy of an id is a stub with only id and title, it overwrites the full copy and you lose the price, currency, and date. The other classic miss is `record['offers'][0]['price']` with no guard, which raises KeyError when a source omits offers and IndexError when offers is an empty list. --- **Naive** out = {} for r in records: out[r['id']] = { 'price': float(r['offers'][0]['price']), ... } Crashes on a missing 'id' key, a source that uses 'sku', an absent 'offers', a price string with a '$', and quietly drops the fuller copy on the second write. **Correct** Pull each field through a key-fallback helper, guard the offers index with isinstance, coerce the price and date, and replace a stored id only when the new copy has strictly more non-null fields. Resilient to new key names and missing paths, and it keeps the most complete record. --- ## Common follow-up questions - A new source site introduces `productName` for the title and nests the date under `metadata.datePosted`. How does your code change? _(Tests whether the extraction is data-driven (add a key to a fallback list, add one path helper) rather than hard-coded, which is the resilience the interview follow-up probed.)_ - Two copies of the same id have the exact same number of populated fields but different values. Which wins, and how would you make that rule explicit? _(Tests tie-breaking policy: strict-improvement keeps first-seen, but a real pipeline might prefer the most recently scraped or a per-field merge.)_ - The crawl no longer fits in memory as one list. How do you flatten and dedup a stream of tens of millions of records? _(Tests scaling: a streaming reduce keyed by id, or a group-by-id in Spark with a completeness-max reducer, keeping the transform associative.)_ ## Related - [All practice problems](https://datadriven.io/problems) - [Mock interview mode](https://datadriven.io/interview/the-fuller-copy) - [Python Interview Questions](https://datadriven.io/python-interview-questions) - [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep) - [Daily Challenge](https://datadriven.io/daily) --- Source: DataDriven (https://datadriven.io). 100% free data engineering interview prep. Live code execution against Postgres 16, Python 3.11, and Spark sandboxes. No paywall, no premium tier, no signup gate.