# Against the Current

> Too big to load. Read what you can.

Canonical URL: <https://datadriven.io/problems/against_the_current>

Domain: Python · Difficulty: hard · Seniority: L5

## Problem

A nightly export hands you a CSV log far too large to fit in memory, delivered as an iterable that yields one line at a time with the header row first. Given a `column_name`, return the sum of that column's numeric values as a float. Rows whose value in that column is missing or does not parse as a number are skipped, and if `column_name` is not among the header cells the sum is `0.0`.

## Worked solution and explanation

### What this problem is really about

This looks like a column sum, but it is really a test of whether you can resist the one instinct that crashes the job. Almost everyone reaches for `list(lines)` or `pandas.read_csv()` to get random access to the rows, and on a multi-gigabyte export that line is an OOM kill before a single number is added. The whole problem is staying lazy: `csv.reader` pulls one line at a time, so the trick is feeding `lines` straight into it and never holding more than a running total in memory.

> **Trick to solving**
>
> Pass `lines` directly into `csv.reader`. It is an iterator adapter, not a loader, so it advances the underlying generator one row per `next()` and you keep O(1) memory no matter how large the file is.

---

### Working through it

#### Step 1: Wrap the iterable in a streaming reader

Hand `lines` to `csv.reader` so it consumes the iterable lazily, row by row. The moment you bind it to a list comprehension or call `list()`, you have lost the only thing this problem grades.

#### Step 2: Resolve the column index once

Pull the first row as the header and locate the target with `header.index(column_name)`. If it raises `ValueError`, the column does not exist and there is nothing to sum, so return 0.0 right away. Guard the empty-input case too: `next(reader)` on an exhausted iterator raises `StopIteration`.

#### Step 3: Convert defensively and accumulate

For each remaining row, grab the value at the column index and convert it. A short row, a blank cell, or a non-numeric string should silently skip, not abort the stream, so wrap the `float()` in a try/except and only add on success.

---

### The solution

**Streaming CSV sum with row-by-row processing**

```python
import csv

def sum_column(lines, column_name):
    reader = csv.reader(lines)
    try:
        header = next(reader)
    except StopIteration:
        return 0.0
    try:
        idx = header.index(column_name)
    except ValueError:
        return 0.0
    total = 0.0
    for row in reader:
        if idx >= len(row):
            continue
        try:
            total += float(row[idx])
        except (ValueError, TypeError):
            continue
    return total
```

> **Time and space complexity**
>
> **Time:** O(n) in the number of rows, each touched exactly once.
> 
> **Space:** O(1) beyond a single line buffer. Only the header and a running total survive between iterations, never the dataset.

> **Interviewers watch for**
>
> The tell is whether your hands reach for `list(lines)`, a full comprehension over the rows, or `pandas.read_csv` without chunking. Any of those signals you have not internalized the memory constraint, and on a 5GB file it is the difference between a job that finishes and one that gets OOM-killed.

> **Common pitfall**
>
> Forgetting that real exports are dirty: a blank cell, a short row, or a stray `N/A` will throw on `float()` and kill the entire pass if you do not catch it. One bad row should cost one skipped row, not the whole sum.

---

## Common follow-up questions

- What if the lines come from a gzip-compressed file? _(Tests wrapping a file with gzip.open and passing the resulting line iterator in.)_
- How would you process the input in parallel? _(Tests splitting by byte offset or using multiprocessing.Pool with chunk boundaries.)_
- What if the CSV uses a non-comma delimiter? _(Tests passing `delimiter` to `csv.reader`.)_
- How would you compute both sum and count in a single pass? _(Tests maintaining two accumulators in the same loop.)_

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/against_the_current)
- [Python Interview Questions](https://datadriven.io/python-interview-questions)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.