The Gaps Between

A hard Python interview practice problem on DataDriven. Write the Python and run it against test cases, with instant feedback.

Domain
Python
Difficulty
hard
Seniority
L4

Problem

A data-quality job profiles raw record exports before they load into the warehouse, and the records are dicts that don't all carry the same columns. For every column that appears in any record, report how many records hold a real value versus how many are blank, where blank means the value is None or the column is absent from that record entirely. Each column maps to its own null_count and non_null_count, and an empty batch produces an empty result.

The data is not as clean as it looks.

Example 1

Input:

records = [{"email":null,"user_id":1},{"email":"ab@x.io","user_id":null},{"email":"cd@x.io","user_id":3}]

Output:

{"email":{"null_count":1,"non_null_count":2},"user_id":{"null_count":1,"non_null_count":2}}

Example 2

Input:

records = [{"score":10},{"grade":"A","score":null}]

Output:

{"grade":{"null_count":1,"non_null_count":1},"score":{"null_count":1,"non_null_count":1}}

Worked solution and explanation

What this really tests

This is a union-of-keys problem wearing a data-quality costume. Adding up the counts per column is the easy part; what separates candidates is realizing the column set is never handed to you. The records are ragged: a column can debut in record 50 and appear nowhere else, and an absent key is itself a null signal. Derive your columns from the first record and you will silently drop every column that shows up later, reporting a clean bill of health on data you never actually inspected.

Break it into three decisions

Step 1: Discover the full column set first

Sweep every record and union their keys. This is the move people skip, and it is the whole problem: the first record is not authoritative about which columns exist.

Step 2: Count against the full column set

For each record and each known column, read the value with record.get(col) so a missing key resolves to None instead of raising. Compare with is None, not a truthiness check, so 0 and the empty string stay on the non-null side.

Step 3: Let missing keys fall out as nulls

Because you iterate the known columns rather than the keys each record happens to have, a record that lacks a column automatically scores a null for it. No special case needed.

The solution

Per-column null profiling over the unioned column set
def column_stats(records):
    columns = set()
    for record in records:
        columns.update(record)
    result = {col: {'null_count': 0, 'non_null_count': 0} for col in columns}
    for record in records:
        for col in columns:
            if record.get(col) is None:
                result[col]['null_count'] += 1
            else:
                result[col]['non_null_count'] += 1
    return result

Common follow-up questions