Storage Architecture

Concepts covered: paMedallion, paPartitioning, paColumnarVsRow, paTableFormats, paSmallFiles, paDataLake

Saying 'I would store it in S3' carries no information. The interviewer needs to hear the organizational model, the file format, the table format, and the partitioning key, because each of those decisions carries a cost that scales with the data. Medallion Architecture Medallion organizes the lake into 3 quality tiers. Bronze holds raw data exactly as ingested, byte-for-byte, including duplicates and malformed records. Silver holds cleaned, typed, and deduplicated records. Gold holds business-ready aggregates shaped for a specific consumer. The critical property is that bronze is immutable. If a transformation bug corrupts gold, the fix is to re-run the transform on bronze without re-extracting from the source. That rebuild capability is the entire reason the layers exist, and naming it un

About This Interactive Section

This section is part of the Design a Pipeline: Intermediate lesson on DataDriven, a free data engineering interview prep platform. Each section includes explanations, worked examples, and hands-on code challenges that execute in real time. SQL queries run against a live database. Python runs in a sandboxed Docker container. Data modeling problems validate against interactive schema canvases. All content is framed around what data engineering interviewers actually test at companies like Meta, Google, Amazon, Netflix, Stripe, and Databricks.

How DataDriven Lessons Work

DataDriven combines four interview rounds (SQL, Python, Data Modeling, Pipeline Architecture) with adaptive difficulty and spaced repetition. Easy problems get harder as you improve. Weak concepts resurface until you master them. Your readiness score tracks progress across every topic interviewers test. Every lesson section ends with problems you solve by writing and running real code, not by picking multiple-choice answers.