# Footprints in the Feed

> Every trade left a footprint.

Canonical URL: <https://datadriven.io/problems/footprints_in_the_feed>

Domain: Python · Difficulty: medium · Seniority: L4

## Problem

You're rolling up raw feed-handler log lines in `lines`, each a CSV row of `date,process,host,log_message,bytes`, where the `process` name carries the exchange as its leading token (so `nyse_feed_handler` is an NYSE process and `nasdaq_market_data` is a NASDAQ one). Roll the byte counts up two ways in one pass: total bytes per day, and total bytes per exchange within each day. Return both rollups together, the per-day totals first.

## Worked solution and explanation

### What this problem really is

This is a two-dimensional rollup wearing a log-parsing costume. The parsing (split a comma row, grab the token before the first underscore) is the part everyone gets right. The skill actually being probed: can you accumulate a per-day total AND a per-exchange-within-day total in the SAME pass, where the second result is a dict of dicts you grow lazily as new (date, exchange) pairs show up. The trap is quietly expensive: candidates build one accumulator, or roll up only by day, then discover they have collapsed the exchange dimension into a single sum and cannot get it back. Once bytes from `nyse` and `nasdaq` land in the same per-day number, no later step can split them apart.

---

### Break down the requirements

#### Step 1: Parse each CSV line into fields

Split on comma to get date, process, host, log_message, and bytes. Convert the bytes field with `int(...)`. The host and log_message columns are noise for this task; you only touch positions 0, 1, and 4.

#### Step 2: Extract the exchange name from the process field

The exchange name is the portion before the first underscore in the process column: `process.split('_')[0]`. Note that a process with no underscore (like `cme`) returns the whole string unchanged, which is exactly the behavior you want.

#### Step 3: Accumulate bytes by day and by exchange-day

Keep two dicts side by side and update BOTH on every line, not one then the other in a second pass. `daily_totals[date]` accumulates a scalar; `exchange_daily[date]` is itself a dict that must exist before you index into it, so initialize it the first time you see a new date. Return them in a list in that exact order: `[daily_totals, exchange_daily]`.

---

### The solution

**CSV split with dual-level aggregation**

```python
def aggregate_trades(lines: list) -> list:
    daily_totals = {}
    exchange_daily = {}
    for line in lines:
        parts = line.split(",")
        date = parts[0]
        process = parts[1]
        bytes_val = int(parts[4])
        exchange = process.split("_")[0]
        daily_totals[date] = daily_totals.get(date, 0) + bytes_val
        if date not in exchange_daily:
            exchange_daily[date] = {}
        exchange_daily[date][exchange] = exchange_daily[date].get(exchange, 0) + bytes_val
    return [daily_totals, exchange_daily]
```

*One pass, both rollups updated together so the exchange dimension is never collapsed.*

> **The defaultdict shortcut**
>
> `collections.defaultdict(int)` for the daily totals and `defaultdict(lambda: defaultdict(int))` for the nested one removes the `.get(..., 0)` and the `if date not in` guard entirely. It reads cleaner, but remember the grader compares against plain dicts, so a defaultdict comparing equal is fine while returning it directly with lingering default factories is also fine here since equality ignores the factory.

> **Time and Space Complexity**
>
> **Time:** O(n) where n is the number of log lines. Each line is parsed and aggregated in constant time.
> 
> **Space:** O(d * e) where d is the number of distinct dates and e is the number of distinct exchanges.

> **Interviewers watch for**
>
> Return a `list` `[daily_totals, exchange_daily]`, not a tuple, and in that exact order: daily totals first, exchange-by-day second. Graders that compare types strictly mark a tuple wrong even when the contents match. The visible empty-input case pins this shape for you: `[]` comes back as `[{}, {}]`, two empty dicts, not one.

> **Common pitfall**
>
> Trying to derive the exchange breakdown after the fact from `daily_totals`. Once a day's bytes are summed into a single number, the per-exchange contributions are gone. The only way to have both is to accumulate both in the same loop.

---

## Common follow-up questions

- How would you handle quoted CSV fields containing commas? _(Tests using the `csv` module instead of naive `split(',')`.)_
- Does your answer change if logs arrive out of chronological order? _(Tests whether the candidate realizes dict aggregation is order-independent.)_
- How would you process a 10GB log file? _(Tests streaming line-by-line processing instead of loading the entire file.)_
- What if the process field had no underscore? _(Tests defensive parsing with a fallback when `split('_')` yields only one part.)_

## Related

- [All practice problems](https://datadriven.io/problems)
- [Mock interview mode](https://datadriven.io/interview/footprints_in_the_feed)
- [Python Interview Questions](https://datadriven.io/python-interview-questions)
- [Data Engineering Interview Prep Guide](https://datadriven.io/data-engineer-interview-prep)
- [Daily Challenge](https://datadriven.io/daily)

---

Source: DataDriven (https://datadriven.io). DataDriven is the data engineering interview community. Live code execution in SQL, Python, and Spark sandboxes. Every feature is open to every member.