Against the Current

A hard Python interview practice problem on DataDriven. Write the Python and run it against test cases, with instant feedback.

Domain
Python
Difficulty
hard
Seniority
L5
Asked in
Deloitte

Problem

A nightly export hands you a CSV log far too large to fit in memory, delivered as an iterable that yields one line at a time with the header row first. Given a column_name, return the sum of that column's numeric values as a float. Rows whose value in that column is missing or does not parse as a number are skipped, and if column_name is not among the header cells the sum is 0.0.

Too big to load. Read what you can.

Example 1

Input:

lines = ["a,b,c","1,2,3","4,5,6"], column_name = "b"

Output:

7

Example 2

Input:

lines = ["amount,note","1.5,ok","xx,bad","2.25,ok"], column_name = "amount"

Output:

3.75

Worked solution and explanation

What this problem is really about

This looks like a column sum, but it is really a test of whether you can resist the one instinct that crashes the job. Almost everyone reaches for list(lines) or pandas.read_csv() to get random access to the rows, and on a multi-gigabyte export that line is an OOM kill before a single number is added. The whole problem is staying lazy: csv.reader pulls one line at a time, so the trick is feeding lines straight into it and never holding more than a running total in memory.

Working through it

Step 1: Wrap the iterable in a streaming reader

Hand lines to csv.reader so it consumes the iterable lazily, row by row. The moment you bind it to a list comprehension or call list(), you have lost the only thing this problem grades.

Step 2: Resolve the column index once

Pull the first row as the header and locate the target with header.index(column_name). If it raises ValueError, the column does not exist and there is nothing to sum, so return 0.0 right away. Guard the empty-input case too: next(reader) on an exhausted iterator raises StopIteration.

Step 3: Convert defensively and accumulate

For each remaining row, grab the value at the column index and convert it. A short row, a blank cell, or a non-numeric string should silently skip, not abort the stream, so wrap the float() in a try/except and only add on success.

The solution

Streaming CSV sum with row-by-row processing
import csv

def sum_column(lines, column_name):
    reader = csv.reader(lines)
    try:
        header = next(reader)
    except StopIteration:
        return 0.0
    try:
        idx = header.index(column_name)
    except ValueError:
        return 0.0
    total = 0.0
    for row in reader:
        if idx >= len(row):
            continue
        try:
            total += float(row[idx])
        except (ValueError, TypeError):
            continue
    return total

Common follow-up questions