The Vital Few

A medium Data Modeling interview practice problem on DataDriven. Write and execute real data modeling code with instant grading.

Domain

Data Modeling

Difficulty

medium

Seniority

L5

Problem

We ingest about 2TB of raw application logs into S3 every day, where debug and info lines vastly outnumber the errors analysts actually care about, so the high-value error records need to live apart from the bulk firehose. Analysts follow a single request as it crosses services and stitch a user's activity together within a session, starting from either an ordinary log line or an error, so the request and session identifiers have to ride on both. Errors are grouped by a canonical category instead of raw message text, and error rates get sliced by the team that owns each service and its tier, so keep that ownership one hop away.

Summary

Two terabytes a day, and the lines that matter are a rounding error in the noise.

Practice This Problem

Solve this Data Modeling problem with real code execution. DataDriven runs your solution and grades it automatically.