Learn SQL, Python, and Data Modeling Interactively

Read any DAG file and say what it describes, which tasks run side by side, and what its graph and grid will show.

Topics: What Airflow Is For, Where Airflow Fits, Tasks in a DAG File, The TaskFlow Decorators, Reading the Graph and Grid

What the dag processor does with your file every 30 seconds, and why that decides whether your DAG shows up at all.

Topics: The Parse Loop, Why a DAG Goes Missing, Operators from Providers, Testing a DAG Locally, Grouping Tasks

One task that becomes forty, one file that becomes fourteen DAGs, and the line that task code cannot cross.

Topics: Dynamic Task Mapping, DAGs Built from Config, DAG Versions and Bundles, The Task SDK Boundary, Writing a Custom Operator

Every scheduled run is named for a window that has already closed; learn to predict which runs exist at any moment.

Topics: A Run Covers a Slice of Time, Schedule Strings and Presets, start_date and the First Run, Catchup On and Off, Triggering a Run by Hand

A run's logical date is its name and its window is what it reads; learn which timetable gives which and what the clock does to both.

Topics: What the Logical Date Names, Two Timetables for One Cron, Backfilling a Date Range, Time Zones and Daylight Saving, Fixed Intervals vs Clock Times

Schedules driven by data events, business calendars, prior runs, and what happens when you change one in production.

Topics: Asset-Triggered DAGs, Asset Conditions and Aliases, Custom Timetables, Runs That Wait on Past Runs, Changing a Live Schedule

One task instance, from scheduled to done: retries, spreading failures, clears, and the log every try leaves behind.

Topics: The Life of a Task Instance, Retries and Retry Delay, Upstream Failed and Skipped, Clearing Tasks to Rerun, Logs, Tries, and Attempts

Trigger rules, branches, callbacks and timeouts: how a task decides to run, skip, page or stop.

Topics: Trigger Rules, Branching and the Skipped Join, Short-Circuits and LatestOnly, Callbacks on Failure, Timeouts and Hung Tasks

Backoff, teardowns, approvals, killed processes and late runs: what Airflow records when a night goes wrong.

Topics: Retry Policies and Backoff, Setup and Teardown Tasks, Human Approval Steps, Deaths Without a Log, Deadline Alerts

Airflow issues the commands; the rows stay where they live, and tasks pass only names, counts and dates.

Topics: Commands, Not Data, Passing Values with XCom, Connections and Hooks, Variables and Params, Templated Fields

How a table says which row is which, how tables point at each other, and how to mint keys that never collide.

Topics: The Problem of Identity, Primary Keys: Data Identity, Foreign Keys, Composite Keys, Key Generation Strategies

Every column's type is a contract with the future; pick the one that cannot store a wrong answer.

Topics: The FLOAT Money Bug, String Types & Platform Traps, Temporal Types & DST, ENUM Traps, Type Review Framework

A relationship is a column that points at another table's rows, and every join you will ever write follows one.

Topics: What Are Relationships?, Cardinality Explained, Required vs Optional, Self-Referential Tables, Complex Patterns

Store each fact in exactly one place, so an update can never leave two copies disagreeing.

Topics: Data Gets Out of Sync, First Normal Form (1NF), Second Normal Form (2NF), Third Normal Form (3NF), Identifying Normal Form

Three normal forms past 3NF, then the rules for breaking all of them on purpose.

Topics: Boyce-Codd Normal Form, Fourth Normal Form (4NF), Fifth Normal Form (5NF), Strategic Denormalization, Denormalization Patterns

One fact table in the centre, dimensions around it, and the grain decision that makes every number add up.

Topics: The Star Schema, Types of Fact Tables, Types of Dimensions, Defining the Grain, Surrogate Keys

Put a list, a record or a bag of key-value pairs inside one column, and know what it costs to read back.

Topics: The Nesting Decision, STRUCT: Embedded Objects, ARRAY: Ordered Collections, MAP: Dynamic Key-Value Pairs, Columnar Storage & Nesting

Model what happened as an append-only log of facts, rebuild state from it, and handle events that arrive late.

Topics: Event-Driven Architecture, Immutable Append-Only Logs, Event Sourcing, Clickstream Modeling, Handling Late-Arriving Data

Compute the GROUP BY once, store the result, and let every dashboard read 27,000 rows instead of 2 billion.

Topics: Why Pre-Aggregate?, Metric Types and Additivity, OLAP Cubes & Rollups, Granularity Design, Refresh Strategies for Aggregates

Five shapes a warehouse can take, what each one makes cheap, and the requirement that breaks it.

Topics: Medallion Architecture, Data Vault, One Big Table (OBT), Semantic Layers, Pipeline DAG Design

A model is one SELECT in a file; dbt turns it into a table or a view, in the order the project requires.

Topics: What dbt Is For, Where dbt Fits, Models Over Sources, View or Table, ref() Orders the Build

Layer a dbt project so each model has one job, one grain, and the right place to live.

Topics: Staging, Intermediate, Marts, One Grain per Model, Seeds for Small Lookups, Ephemeral Models, Snapshots Keep History

Promise a model's shape, change it without breaking readers, record vanished rows, and clean up what dbt leaves behind.

Topics: Model Contracts, Model Versions, Snapshot Strategies, Tables dbt Left Behind, Refactoring a Legacy Query

Every dbt test is a query that hunts for bad rows, and a build halts the models that would have read them.

Topics: A Test Is a Query, The Four Generic Tests, Properties in YAML, Failing Tests Stop Children, Source Freshness

Decide which test failures stop a build, write rules no generic test states, and keep the rows that failed.

Topics: Warn or Error, Singular Tests, Storing Failing Rows, Testing a Compound Key, Deciding What to Test

Prove a model's logic on rows you wrote, tolerate the dirt you know about, and build on nothing before building on everything.

Topics: Unit Tests on Fixed Inputs, Fixtures and Overrides, Failure Thresholds, Empty Builds as a Check, A Test Plan per Layer

dbt turns your Jinja into SQL before the warehouse sees a line, and you can read exactly what it sent.

Topics: Jinja Renders First, Reading Compiled SQL, Set, If, and For, Where a Config Comes From, Compile vs Database Errors

Write SQL once as a macro, borrow more from a package, turn checks into reusable tests, and feed values in from outside.

Topics: Writing a Macro, Packages and dbt_utils, Writing a Generic Test, Macros That Return Values, vars and env_var

Jinja that reads the warehouse, SQL that runs outside the graph, and the discipline of knowing when to write neither.

Topics: Parse Time vs Run Time, Querying from Jinja, Hooks and Operations, Custom Materializations, When Not to Write a Macro

Add the rows that arrived since the last build, instead of rebuilding a table from its first row every time.

Topics: Rebuild or Increment, is_incremental() and this, Filtering Only New Rows, The First Run Builds It All, Full Refresh

One append-only log that every system writes to once and every reader reads at its own pace

Topics: What Kafka Is For, Where Kafka Fits, The Partitioned Log, What a Record Carries, Reading Does Not Delete

Segments, retention, compaction, tombstones and partition counts: what a partition keeps, drops and cannot undo.

Topics: Segments on Disk, Retention by Time and Size, Compacted Topics, Tombstones and Deletes, Choosing a Partition Count

Compaction timing, two clocks, future stamps, tiered segments and oversized records: the log's advanced mechanics.

Topics: When Compaction Runs, Two Kinds of Timestamp, Timestamps from the Future, Tiered Storage, Large Messages

Every partition lives on several brokers, 1 of them takes the writes, and the rest stand ready to take over.

Topics: Brokers Hold Replicas, Leaders and Followers, In-Sync Replicas, Replication Factor, When a Broker Dies

Follow a record from the leader's disk to a reader's screen, and learn what the cluster does when a copy, a broker or a whole zone disappears.

Topics: The High Watermark, Minimum In-Sync Replicas, Followers That Fall Behind, Electing a New Leader, Replicas Across Racks

What a cluster trades away for availability, where its metadata lives, and how partitions travel between brokers, racks and regions

Topics: Unclean Leader Election, KRaft Controllers, Moving Partitions, Mirroring to Another Cluster, Reading the Nearest Replica

Follow one record from send() to the partition's log, and learn which producer setting decides each step.

Topics: Sending a Record, Keys Pick Partitions, When a Send Counts, Send Errors and Retries, Hot Keys and Skew

Batching, compression, idempotence, ordering and acks: the producer settings that decide cost, duplicates and loss.

Topics: Batching and Linger, Compression on the Wire, Idempotent Producers, Ordering Under Retries, The Durability Contract

Atomic writes across partitions, fencing the writer you replaced, and the 2 ways a key quietly changes partition.

Topics: Transactions, Fencing Zombie Producers, One Key, Two Partitioners, Adding Partitions Remaps Keys, Full Buffers and Blocking

Follow a consumer from its first poll to its committed offset, and read how far behind its group has fallen.

Topics: The Poll Loop, Groups Split Partitions, Committing an Offset, Where a New Group Starts, Reading Consumer Lag

Why pipelines exist, the 4 roles every one is built from, how to read one, and when a query would do instead.

Topics: Why Pipelines Exist, The Four Roles in Any Pipeline, Reading a Pipeline Left to Right, A First End-to-End Pipeline, When a Pipeline Is Not Needed

How a pipeline grows from one chain into layers, forks and joins, and how to read what the grown-up diagram says.

Topics: Many Sources, One Curated Layer, ETL vs ELT, Steps That Merge Many Inputs, Reading a Real Pipeline Diagram, One Source, Two Consumers

A pipeline is a product: name its consumer, write its contract, plan its retirement, then untangle the rest.

Topics: Pipelines as Products, The Cross-Cutting Undercurrents, When to Split, When to Merge, Build vs Buy at Each Layer, Redesigning a Tangled Graph

Two rhythms move data: wait and process the pile, or handle each record the moment it lands.

Topics: Two Ways Data Can Move, Batch: Picture, Rhythm, Example, Streaming: Picture, Rhythm, Example, What Real-Time Actually Means, Picking Batch or Streaming

Measure latency and throughput as 2 separate numbers, then buy exactly as much freshness as the consumer can use.

Topics: Latency vs Throughput Tradeoff, Micro-Batch: The Middle Ground, Why Streaming Costs More, Stateful vs Stateless Transforms, When Batch Outgrows Itself

Why Lambda split the pipeline in two, how Kappa rejoined it, and what each node's freshness tier really costs.

Topics: Lambda Architecture, Kappa: Stream Only, Batch Replay, Unified Engines: Where Lines Blur, Per-Node Freshness Tier Analysis, Lambda to Kappa Worked Example

One set of orders, three shapes on disk: why the database, the warehouse and the lake each win a different read.

Topics: Storage Is Not Just the Database, The Data Warehouse, The Data Lake, The Operational Database, Picking the Right Storage Shape

Columnar files, partitions, codecs and pushable filters: how they stack to cut a 10TB scan to 100GB.

Topics: Columnar Versus Row Storage, Partitioning and File Pruning, Compression: Bytes Versus CPU, Predicate Pushdown, Turning a 10TB Scan Into 100GB

Object storage plus a metadata layer: how lakehouse tables get atomic commits, time travel and cheap schema change

Topics: The Lakehouse: ACID on Object, Snapshot Isolation and Time Travel, Schema Evolution Without Rewrites, The Small Files Problem, Choosing Storage Across Workloads

Cron starts jobs on a clock; an orchestrator starts them when the jobs they depend on have succeeded.

Topics: Why Cron Is Not an Orchestrator, The DAG: Tasks, Edges, No Cycles, What an Orchestrator Does, The Major Orchestrators by Name, First DAG: 3 Tasks, 1 Schedule

Names, values, the 4 types on every line, printing them, combining them, and the comments that explain why.

Topics: Variables and Assignment, Data Types, Print Statements, Basic Operators, Comments

Blocks, imports, input, tracebacks and the shell: the 5 habits that turn lines of Python into a program you can run.

Topics: Indentation and Code Blocks, Importing Modules, Reading Input From the User, Reading a Traceback, Trying Code in the Interactive Shell

Recover from bad input, stream files line by line, bundle data into classes, and close every resource you open.

Topics: Handling Errors With try and except, Reading and Writing Files, Defining a Class, Generators, Context Managers

Values live in memory, names point at them, and an operator only works when the types on both sides agree.

Topics: How Computers Store Data, Variables and Naming, Assignment vs. Equality, Mixing Types in One Expression, Operators and Readability

Convert text to numbers, compare without surprises, update in place, and tell None apart from zero.

Topics: Type Conversions, Comparison Operators, Augmented Assignment, Multiple Assignment, None and Identity

Predict what Python's trickiest expressions return: floats, and/or, truthiness, bit flags and the walrus.

Topics: Floating-Point Surprises, Short-circuit Evaluation, Truthy and Falsy Values, Bitwise Operators on Integers, Walrus Operator (:=)

Write conditions, branch on them, and trace exactly which lines Python runs.

Topics: The if Statement, Branching with if-else, Chaining if-elif-else, Combining with and/or, Execution Flow

Flatten nested checks, test ranges in one expression, and dispatch on values and types without a staircase of ifs.

Topics: Guard Clauses, Chained Comparisons, Pattern Matching with match-case, Conditional Assignment, Branching on Types With isinstance

Prune checks that never matter, negate conditions correctly, and turn branching rules into states and lookup tables.

Topics: Removing Redundant Conditions, De Morgan's Laws, State Machine Patterns, Dict-Based Dispatch, Decision Table Lookups

Make Python repeat itself: walk lists and strings, count with range, loop while a condition holds, and exit on demand.

Topics: Iterating with for Loops, range() Function, Loops with while, Using break and continue, Loop Variable Scope

What the driver plans, what the executors run, and why a 100-core cluster can sit 96 cores idle.

Topics: Driver, Executors, and Tasks, Partitions and Parallelism, Transformations vs Actions, Cores and Slots, A Job's Life, End to End

Count stages before you run, read task counts against slots, and know where the driver lives before you tune.

Topics: Job, Stage, Task, Why Stages Exist, Reading Parallelism, Where the Driver Lives, The spark-submit Config Surface

Two schedulers, one driver: how Spark places tasks, retries them, duplicates stragglers, and where it all jams.

Topics: DAGScheduler vs TaskScheduler, Task Failure and Retry, Speculative Execution, The Driver as Bottleneck, Locality and Scheduling Delay

Spark runs nothing until an action asks for a result, which explains its late errors, its speed and its repeated work.

Topics: Errors That Surface at the Action, Why Laziness Makes Spark Fast, The Action Catalog, The collect() Trap, Re-Execution on Every Action

See the graph Spark builds before it runs, catch the jobs that fire early, and stop a write from eating its own input.

Topics: The DAG as a Graph, Jobs That Run Before an Action, Overwriting What You Read, Pipelining Narrow Operations, DAG vs Lineage

Why a loop makes Spark replay every earlier round, and how checkpoints and careful side effects stop it.

Topics: Lineage That Grows in Loops, The Recompute Cost, Checkpointing to Cut Lineage, Local vs Reliable Checkpoints, Side Effects Inside Transformations

Narrow calls stay on their partition; wide calls move every row. Learn to tell them apart and count what moving costs.

Topics: Narrow Transformations, Wide Transformations, What a Shuffle Actually Is, Why Wide Costs More Than Narrow, Spotting Shuffles in Your Code

Follow a shuffle's bytes from a map task's local disk to a reducer's fetch, and learn when Spark can skip moving them.

Topics: The Shuffle Write, The Shuffle Read, Combining Before the Shuffle, Hash vs Range Partitioning, Reusing Shuffle Output

What a shuffle writes to disk, who serves it, what it costs, and how to delete or tune it.

Topics: Sort-Based Shuffle Internals, The External Shuffle Service, Pricing a Shuffle, Eliminating a Shuffle, The Shuffle Tuning Knobs

Spark rewrites every DataFrame query before it runs. See what it changes, what blocks it, and how to read the result.

Topics: DataFrames over RDDs, Your Query Gets Rewritten, The RDD Escape Hatch, DataFrame, Dataset, and RDD, Seeing It with explain()

Read a table, name the columns you want, relabel them, and compute new values: the 4 moves inside every query.

Topics: Tables, rows, and columns, SELECT and FROM basics, Selecting all columns (*), AS aliases for columns, Expressions in SELECT

Name your steps, know the order they run in, then nest, stack and slice result sets on purpose.

Topics: CTEs (WITH clause), Query Execution Order, Subqueries for temp results, UNION and UNION ALL, ORDER BY and LIMIT

Per-row subqueries, set subtraction and overlap, arrays turned into rows, and queries that need no table at all.

Topics: Correlated subqueries, EXCEPT and EXCEPT ALL, INTERSECT and INTERSECT ALL, UNNEST for arrays, SELECT Without FROM

Why a column's type decides what the database can add, compare and sort, and how CAST moves a value between types.

Topics: Why data types matter, INTEGER for whole numbers, STRING types (VARCHAR), BOOLEAN for true/false, CAST for type conversion

Exact money, pinned moments, and conversions that survive bad input.

Topics: BOOLEAN and NULL logic, DECIMAL precision and scale, TIMESTAMP vs TIMESTAMPTZ, Time zones and UTC handling, TRY_CAST and implicit casts

Choose types that fit billions of rows, price their bytes, query maps and nested arrays, and survive bad values.

Topics: Type optimization at scale, Storage calculations and VARCHAR, MAP data type for key-value pairs, Accessing nested data (UNNEST), Compression and error handling

WHERE turns a whole table into the rows a question asks for, one test per row.

Topics: WHERE clause for filtering rows, Equals and not equals (=, !=), Comparison operators (<, >), IN and NOT IN for list matching, AND for combining conditions

Branch, group, pattern-match, page and range-filter your rows, and predict exactly which ones come back.

Topics: OR and CASE expressions, Operator precedence, LIKE for pattern matching, LIMIT and OFFSET for pagination, BETWEEN for range filtering

Test for related rows, dodge the NULL inside NOT IN, and validate messy text with regular expressions.

Topics: Correlated subqueries (EXISTS), NOT EXISTS for missing rows, NOT IN vs NULL gotchas, REGEXP_LIKE patterns, Regex operators and patterns

Collapse rows into the per-group numbers a report needs, and know exactly what each number counts.

Topics: GROUP BY for categorizing data, COUNT variations (*,col,DISTINCT), SUM and AVG calculations, MIN and MAX for extremes, HAVING for filtered groups