Top 15 Data Engineering Projects

From First Pipeline to Production

15 data engineering projects, ordered so each one builds on the last. Every entry links a verified public GitHub repo, with its star count, a difficulty rating, a time estimate, and an open dataset. 11 of the 15 run on a laptop in Docker with no cloud account. The list starts with a dbt project you can finish in an afternoon and ends with Terraform blueprints for Spark on Kubernetes. Pick the entry point that matches what you already know; the decision table below maps 5 common starting situations to a first project.

Last updated: Proudly published by: Jeff Wahl14 min read
15
Projects, all with public repos
11/15
Run locally for $0
44.3k★
Largest repo on the list
38
Tools across the stack

The 15 projects at a glance

Ordered as a build path: each tier assumes the skills of the one before it. Star counts checked August 2026. Click through to any project below.

#ProjectDifficultyTimeCore stackCost
01Jaffle Shop on DuckDBBeginner1 afternoondbt · DuckDB · SQL$0
02Incremental API Pipelines with dltBeginner1 weekendPython · dlt · DuckDB$0
03Data Engineering Practice ProblemsBeginner+2-4 weeks (10 exercises)Python · Docker · PySpark$0
04Batch Pipeline with Airflow, Spark & DuckDBIntermediate1-2 weeksAirflow · Spark · DuckDB$0
05Data Engineering Zoomcamp CapstoneBeginner+9 weeks part-timeTerraform · GCP · BigQuery$0 (free tier)
06MDS-in-a-Box: NBA Monte Carlo SimulatorIntermediate~10 hoursDuckDB · dbt · Meltano$0
073 Billion Rows: the NYC Taxi WarehouseIntermediate~2 weeks (+500 GB disk)PostgreSQL · PostGIS · ClickHouse$0
08A Local Delta Lake Lakehouse, Done RightIntermediate+~20 hoursPySpark · Delta Lake · MinIO$0
09Streaming Attribution with Flink & KafkaBeginner+1 weekendFlink · Kafka · Postgres$0
10Streamify: Kafka to Spark to dbt on GCPIntermediate+2-3 weeksKafka · Spark · AirflowCloud billing
11CDC Pipelines with DebeziumIntermediate+1 weekend to 3 weeksDebezium · Kafka · Postgres$0
12The Official Apache Flink Training LabsAdvanced2-4 weeksFlink · Java · Scala$0
13CI-Enforced Data ContractsIntermediate1 weekendPython · dbt · Snowflake$0
14Column-Level Lineage with Marquez & OpenLineageIntermediate1-2 weekendsOpenLineage · Airflow · Spark$0
15A Data Platform on KubernetesAdvanced1 weekend per blueprintTerraform · Kubernetes · AWSCloud billing

Weighted by what interviews ask about

Tool mentions across 1,042 verified data engineer interview rounds on this platform. The projects above were chosen so the high-frequency tools get the most coverage.

  1. Airflow (orchestration)
    14% of rounds

    Covered by projects 4, 5, 10, 14, 15

  2. dbt (transformation)
    9% of rounds

    Covered by projects 1, 5, 6, 10, 13, 14

  3. Snowflake / BigQuery (warehouse)
    8% of rounds

    Covered by projects 2, 5, 10, 13

  4. Spark (distributed compute)
    7% of rounds

    Covered by projects 3, 4, 5, 8, 10, 14, 15

  5. Kafka (streaming)
    5% of rounds

    Covered by projects 9, 10, 11, 13, 15

How to pick your starting point

Match the entry point to what you already know and the roles you want. Every path eventually runs through Tier 2; the branch points are on either side of it.

If your situation is
Pick
Why
You know SQL but have never shipped a pipeline
Projects 1 → 2 → 4
dbt on DuckDB, then an incremental API load, then orchestration. The full extract-model-serve loop in 3 weekends, $0 spend.
You come from software engineering
Projects 4 → 8
You already have Docker and Git. The gap is orchestration and warehouse modeling; these 2 repos cover both and are small enough to finish.
You have 2 months and want 1 recognizable credential
Project 5
The Zoomcamp capstone is peer reviewed, community backed, and one of the few portfolio lines hiring managers already know.
The roles you want name Kafka or Flink
Projects 9 → 11 → 12
A weekend Flink starter, then CDC patterns, then the official training labs. The concepts build on each other in this order.
You are targeting senior or platform roles
Projects 13 → 14 → 15
Contracts, lineage, and infrastructure as code come up in platform loops, and all 3 projects attach to pipelines you built in earlier tiers.

Foundations: prove you can ship a pipeline

4 projects that cover the extract-model-serve loop end to end. All 4 run on a laptop with Docker, cost $0, and produce something you can demo in a screen share.

Tier 1 · Projects 1-4
Project 01 · Analytics Engineering

Jaffle Shop on DuckDB

Beginner1 afternoonLocal & free

dbt Labs' canonical example project, rewired to run against a local DuckDB file. You build staging models, marts, schema tests, and generated docs from 3 seed tables: customers, orders, payments.

Setup takes under a minute and a full build runs in about a second, so you can change a model from view to table to incremental and immediately inspect what dbt created in DuckDB. That feedback loop makes it cheap to work through every core dbt concept in one sitting: sources, staging versus marts, ref() dependency graphs, schema tests, and generated docs.

Interview signal

Jaffle Shop is dbt Labs' own teaching project, and interviewers at dbt shops know it by name. Materialization trade-offs, which this repo lets you try directly, are a standard analytics engineering question.

dbtDuckDBSQL
dbt-labs/jaffle_shop_duckdb~280★DataJaffle Shop seed CSVs (bundled)
Project 02 · API Ingestion

Incremental API Pipelines with dlt

Beginner1 weekendLocal & free

A production-shaped ingestion pipeline in pure Python: pull a paginated REST API, load it incrementally with cursor state, and land typed tables in DuckDB. Then flip one config line and the same pipeline loads Snowflake or BigQuery.

dlt handles pagination, retries, schema inference, and incremental cursor state, and each one is inspectable while the pipeline runs. When the API adds a field mid-load, you can watch the schema evolve and the new column appear in the destination. Nested JSON gets normalized into typed child tables automatically, which shows you what a normalization layer does before you ever have to build one by hand.

Interview signal

Take-home reviewers look for incremental loading, cursor state, and schema-change handling. A one-shot CSV load shows none of those; this project produces all 3 in a weekend.

PythondltDuckDBSnowflake
dlt-hub/dlt~5.7k★DataAny REST API (GitHub, Chess.com, CoinCap)
Project 03 · Take-Home Drills

Data Engineering Practice Problems

Beginner+2-4 weeks (10 exercises)Local & free

10 progressively harder exercises, each with a Dockerfile and tests: flaky HTTP downloads, web scraping, messy JSON to tabular conversion, PySpark aggregations, and validation with Great Expectations.

Each exercise covers one skill real pipelines need: building download URIs programmatically, surviving HTTP failures, parsing scraped pages, flattening awkward JSON into tables. Later exercises solve the same aggregation with PySpark and then with DuckDB and Polars, which is a practical way to learn where a single machine is enough. Everything runs in Docker with tests, so the working habits carry straight into a job.

Interview signal

The exercises resemble the take-home screens companies send at the phone-screen stage. Finishing the set leaves a public commit history of solved, tested problems.

PythonDockerPySparkPostgresGreat Expectations
danielbeach/data-engineering-practice~2.8k★DataOpen data files fetched per exercise
Project 04 · Batch Orchestration

Batch Pipeline with Airflow, Spark & DuckDB

Intermediate1-2 weeksLocal & free

A textbook batch DAG, entirely local: extract from Postgres and MinIO object storage, transform with Spark, load a DuckDB warehouse, and publish a Quarto dashboard, all orchestrated by one Airflow DAG and booted with a single make up.

The repo is small enough to read every line, and the DAG crosses 4 systems: it waits on a sensor, submits a Spark job, loads DuckDB, and renders a dashboard. Working through it teaches idempotent task design and the reason each hop lands in object storage first. The companion write-up walks through each design decision, which matters later when an interviewer asks why the pipeline is shaped this way.

Interview signal

Hiring managers recognize this architecture on sight, and the suggested extensions (swap a component, add a quality gate) turn a known template into your own project.

AirflowSparkDuckDBMinIODocker
josephmachado/beginner_de_project~590★DataSynthetic user purchase events (bundled)Written walkthrough
raw_customersPKidfirst_namelast_nameraw_ordersPKidFKuser_idorder_datestatusraw_paymentsPKidFKorder_idpayment_methodamount
The seed schemaProject 1 starts from these 3 tables. Staging models rename and type them; the customers mart joins all 3 into 1 wide table with lifetime value.
Why it mattersWarehouse interview questions about grain, joins, and slowly changing attributes are variations on this shape. Learning it at 3 tables makes the scaled versions readable.

Warehouse & lakehouse: model data people can trust

Modeling and warehouse work fill most data engineering days. These 4 projects cover dimensional modeling, warehouse loading and cost, and the lakehouse pattern, at sizes from 3 seed tables to 3 billion rows.

Tier 2 · Projects 5-8
Project 05 · End-to-End Capstone

Data Engineering Zoomcamp Capstone

Beginner+9 weeks part-timeFree cloud tier

The best-known free data engineering course on GitHub: 9 modules where you provision GCP with Terraform, orchestrate monthly loads, partition a BigQuery warehouse, layer dbt models on top, and close with a peer-reviewed capstone.

Every module operates on the same taxi dataset, so ingestion decisions surface again weeks later: a file format or partitioning choice from module 2 shows up as BigQuery bytes scanned and dbt model complexity in module 5. The course ends with a capstone you design yourself, reviewed by peers against a public rubric.

Interview signal

A completed Zoomcamp capstone is one of the few portfolio credentials hiring managers have heard of. The cohort leaderboard gives you peer-reviewed proof of work.

TerraformGCPBigQuerydbtSparkKafka
DataTalksClub/data-engineering-zoomcamp~44.3k★DataNYC TLC Trip Records
Project 06 · Modern Data Stack

MDS-in-a-Box: NBA Monte Carlo Simulator

Intermediate~10 hoursLocal & free

The entire modern data stack compressed onto one node: Meltano ingests sports data, dbt models it in DuckDB, a Monte Carlo season simulator runs as SQL, and Evidence renders a versioned BI site deployed by GitHub Actions.

The whole stack (ingestion, transformation, simulation, BI) runs on one machine with no paid services, which demonstrates that each layer is a swappable interface. The Monte Carlo simulator itself is dbt-managed SQL running thousands of season outcomes per build. Evidence renders the results as a static site that GitHub Actions rebuilds on every merge, so the BI layer goes through code review like everything else.

Interview signal

The live demo at mdsinabox.com gives you something clickable to put in front of an interviewer, and the repo covers dbt, DuckDB, and CI in one place.

DuckDBdbtMeltanoEvidenceGitHub Actions
matsonj/nba-monte-carlo~610★DataBasketball Reference + DraftKings oddsLive demo
Project 07 · Warehouse at Scale

3 Billion Rows: the NYC Taxi Warehouse

Intermediate~2 weeks (+500 GB disk)Local & free

Todd Schneider's famous analysis repo: load 3+ billion taxi, Uber, and Lyft trips into PostgreSQL with PostGIS, join them against taxi-zone shapefiles and weather data, and run the same workload on ClickHouse to feel the columnar difference.

At 3 billion rows, a load strategy that worked on a sample takes days on the full table, and an index decision changes a query from minutes to seconds. The repo's 2022 commits, written when the TLC switched its published format from CSV to Parquet, document what surviving an upstream format change looks like in a maintained pipeline. Loading the same data into ClickHouse produces your own row-store versus column-store benchmark numbers.

Interview signal

Few candidates below senior have queried billions of rows. After this project you can quote your own numbers when a system design round turns to warehouse performance.

PostgreSQLPostGISClickHouse
toddwschneider/nyc-taxi-data~2.1k★DataNYC TLC Trip Records, 3B+ trips since 2009
Project 08 · Lakehouse

A Local Delta Lake Lakehouse, Done Right

Intermediate+~20 hoursLocal & free

A genuine lakehouse on your laptop: Delta Lake tables on MinIO object storage, Spark compute, bronze-silver-gold layers ending in dim_customer and fct_orders, with Great Expectations gates between every layer.

Great Expectations checkpoints run between the bronze, silver, and gold layers, so a bad batch fails its gate before it reaches fct_orders. The gold layer is a dimensional model built inside a lakehouse, and the PySpark is organized into tested classes. Delta Lake supplies ACID transactions, schema enforcement, and time travel over plain Parquet files on MinIO; after operating that for a week you can explain why the combination displaced the separate lake-plus-warehouse design.

Interview signal

Delta Lake and lakehouse appear by name in job descriptions, and data quality comes up in nearly every loop. Describing the checkpoint that failed during development, and what you changed, is a strong answer to both.

PySparkDelta LakeMinIOGreat ExpectationsDocker
josephmachado/data_engineering_best_practices~220★DataAdventureWorks orders & customers (bundled)
The lakehouse pattern (project 8, local via Docker)
Source
Bronze
Silver
Gold
Serving
orders + customers
adventureworks
Delta on MinIO
bronze tables
Great Expectations
bronze checks
err:Fail the run
deduped + typed
silver tables
Great Expectations
silver checks
err:Fail the run
dim_customer
gold marts
• fct_orders
reporting
sales mart

Project 8's medallion flow. PySpark reads and writes every layer as Delta tables on MinIO; each promotion passes a Great Expectations checkpoint first.

ACID on object storageDelta Lake brings transactions, time travel, and schema enforcement to plain Parquet files on MinIO. That combination is why the lakehouse displaced the 2-system lake-plus-warehouse design.
Quality as a gateChecks run between layers and block promotion on failure. The operational decision (fail the pipeline versus quarantine and continue) is the part interviewers push on.
Same shape at workSwap MinIO for S3 and this is the medallion architecture on most Databricks and EMR teams. The local version involves identical decisions with zero spend.

Streaming & CDC: move data while it is still an event

Streaming roles pay a premium because few candidates have worked with event-time state. These 4 projects run from a weekend Flink starter to the official Apache training labs.

Tier 3 · Projects 9-12
Project 10 · Streaming Capstone

Streamify: Kafka to Spark to dbt on GCP

Intermediate+2-3 weeksCloud account

A music-streaming analytics platform: Eventsim emits realistic listen events into Kafka, Spark Structured Streaming lands micro-batches in a GCS data lake, hourly dbt jobs build BigQuery marts, and Terraform provisions all of it.

Spark reads Kafka every 2 minutes and lands micro-batches in GCS; dbt models the lake into BigQuery marts once an hour. Most production analytics platforms run this same split of streaming ingestion and batch transformation, and Streamify has you operate both sides plus the boundary between them. Terraform provisions the whole platform, so tearing it down and standing it back up is one command.

Interview signal

Kafka, Spark, Airflow, dbt, and Terraform in one repo covers most of the tools named on a typical mid-level posting. It is also one of the best-known DataTalksClub capstones, so the architecture reads as credible to reviewers who have seen it before.

KafkaSparkAirflowdbtTerraformBigQuery
ankurchavda/streamify~890★DataEventsim clickstream (Million Song seed)
Project 11 · Change Data Capture

CDC Pipelines with Debezium

Intermediate+1 weekend to 3 weeksLocal & free

The official Debezium example collection: stream every change from Postgres, MySQL, or MongoDB transaction logs into Kafka, then work through the pattern examples that matter: outbox, saga, CQRS, and cache invalidation.

Debezium reads the transaction log (Postgres WAL, MySQL binlog) rather than polling tables, which changes how you reason about ordering, initial snapshots, and schema evolution. Spend the most time on the outbox example: a service inserts into an outbox table in the same transaction as its business write, and CDC publishes the event, so there is no dual-write to get wrong. Dual-write consistency comes up in most event-driven system design rounds.

Interview signal

CDC appears on many data platform job descriptions, and this is the official, maintained example collection. Running the outbox pattern gives you working code behind an answer most candidates have only read about.

DebeziumKafkaPostgresMySQLMongoDB
debezium/debezium-examples~1.7k★DataYour own INSERT/UPDATE/DELETE statements
The streaming spine (projects 9-12 share it)
clicks
event producers
• orders
Postgres WAL / MySQL binlog
oltp database
Debezium (project 11)
cdc connector
topics + consumer groups
kafka
Flink / Spark Structured Streaming
stream job
keyed stateGrafana: state size, checkpoints
warehouse / OLAP
analytics sink
Grafana / Data Studio
live dashboard

Kafka decouples producers from consumers; the stream job owns keyed state, windows, and watermarks; Debezium turns the database transaction log into one more event source.

The buffer decouplesKafka lets producers and consumers fail, scale, and deploy independently. Most streaming interview questions are versions of what the buffer buys you.
State is the hard partWindows, watermarks, and TTLs decide correctness under late and out-of-order events. Projects 9 and 12 make those decisions concrete.
CDC feeds the streamReading the transaction log (project 11) turns any OLTP database into an event source without dual-write bugs.

Production & platform: the senior signals

Contracts, lineage, and infrastructure as code come up in senior and platform interviews, and few candidates can show working versions. Each of these 3 attaches to pipelines you built in the earlier tiers.

Tier 4 · Projects 13-15
Project 13 · Data Contracts

CI-Enforced Data Contracts

Intermediate1 weekendLocal & free

Formalize the producer-consumer boundary of any earlier project: declare schema, quality rules, SLAs, and ownership in a YAML contract, run datacontract test against live data, and wire it into CI so breaking changes fail before consumers break.

A contract YAML declares schema, quality rules, SLAs, and ownership for one dataset, and datacontract test executes those rules against the live table. The same file exports to dbt tests and JSON Schema, so every enforcement artifact is generated from one source. Wire it into CI on project 4 or 8 and an upstream column rename fails the pull request instead of paging whoever owns the dashboard.

Interview signal

Data contracts get written about far more often than they get built, and almost no portfolio shows one. A working, CI-enforced contract is a weekend of work here.

PythondbtSnowflakeKafkaCI/CD
datacontract/datacontract-cli~980★DataYour own tables or the bundled examples
Project 14 · Lineage & Observability

Column-Level Lineage with Marquez & OpenLineage

Intermediate1-2 weekendsLocal & free

Run Marquez (from WeWork, reference implementation of OpenLineage), then instrument the Airflow and dbt pipelines you built in earlier projects so every run emits lineage events into a browsable graph.

OpenLineage events are emitted at run time by the orchestrator, which is a different and more reliable approach than parsing SQL after the fact; interviewers notice candidates who know the difference. Working with the event model (runs, jobs, datasets, facets) lets you answer questions like which downstream tables last night's failed job touched, using a graph query. Running the ingestion also shows that metadata collection is itself a pipeline, with its own failure modes.

Interview signal

Impact-analysis questions appear in most data platform loops. A lineage graph of your own DAGs, with a failed run visible in it, is direct evidence you have operated this tooling.

OpenLineageAirflowSparkdbtKubernetes
MarquezProject/marquez~2.3k★DataSeed metadata, then your own Airflow DAGs
Project 15 · Platform Engineering

A Data Platform on Kubernetes

Advanced1 weekend per blueprintCloud account

AWS Labs' blueprint collection for running data infrastructure on EKS: pick one (Spark Operator with YuniKorn, Airflow on EKS, Flink, Kafka via Strimzi), deploy it with Terraform, run a workload, and tear it down.

The Terraform module trees are written by AWS solutions architects and compose VPCs, node groups, autoscaling, and Helm releases the way a platform team would. The Spark blueprint adds the YuniKorn scheduler, so you see how batch jobs from multiple tenants get arbitrated on one cluster. You pay the AWS bill while it runs, which makes spot instances, Karpenter autoscaling, and disciplined teardown part of the exercise.

Interview signal

Airflow and Spark on Kubernetes with Terraform is close to a job description template for platform roles. Deploying and tearing down a blueprint gives you specifics (node sizing, scheduler behavior, what broke) to bring into those interviews.

TerraformKubernetesAWSSparkAirflowFlink
awslabs/data-on-eks~850★DataBlueprint sample workloads (TPC-style benchmarks)

Where the data comes from

Every project above uses an open dataset, a public API, or a bundled generator. No gated platforms, no signup walls. These 6 sources cover nearly any pipeline you want to build next.

The canonical one

NYC TLC Trip Records

3+ billion taxi and rideshare trips since 2009, published monthly as Parquet. The shared dataset of projects 5, 7, and 12, which makes those 3 directly comparable on your resume: same data, 3 architectures. nyc.gov/tlc

Live APIs

Public REST & WebSocket APIs

GitHub, CoinCap, OpenWeather, and transit feeds update continuously, which forces incremental logic instead of a one-shot CSV load. Projects 2 and 6 are built on them. Prefer an API with daily change over any static download.

Synthetic generators

Eventsim, jafgen & friends

Realistic event streams at any volume you want: Eventsim simulates music-listening clickstreams (project 10), jafgen mints years of e-commerce orders. Generators let you demo streaming without production traffic and load-test a pipeline at volumes you choose.

Open catalogs

Kaggle, Hugging Face, data.gov

300k+ government datasets on data.gov, community datasets on Kaggle, and the Hugging Face hub for anything text-shaped. Real public data is messy, and handling its edge cases is where most of a project's interview material comes from.

Your own databases

Change events from OLTP

Projects 11, 13, and 14 need no external dataset at all: the event source is your own INSERT and UPDATE statements, captured from the transaction log or checked against a contract while the database runs.

Replayed history

Batch data replayed as streams

Project 12 replays historical taxi trips as live event streams. Converting a batch dataset into a realistic event source is a skill you will reuse in every streaming demo you ever build, since production traffic is never available for one.

3 common portfolio mistakes

The Myth
More projects means a stronger portfolio.
The Reality
Reviewers spend under a minute before deciding. 2 or 3 finished, documented, tested projects with clear extensions beat 10 half-done clones. Archive everything that is not your best work.
The Myth
It only counts if it runs in the cloud.
The Reality
11 of the 15 projects here run locally in Docker, including a lakehouse and a 3-billion-row warehouse. Hiring managers evaluate the architecture and the trade-offs you can defend, none of which requires a cloud bill. Add 1 cloud deploy when a target role names the provider.
The Myth
Finishing the tutorial is the finish line.
The Reality
Interviewers recognize the popular templates, so they ask what you changed. Swap the data source, add a quality gate, and write down what broke and how you fixed it. That extension work is what the interview conversation will be about.

Turning a repo into interviews

A hiring manager's first pass over a portfolio project lasts about 47 seconds. These 4 steps determine what that first pass finds.

  1. 01

    Extend before you publish

    Swap the data source, add an incremental load, wire in a quality gate or a data contract from project 13. Keep notes on what broke while you did it; the notes turn into interview answers later.
  2. 02

    Write the README for a 47-second scan

    One paragraph on the problem, an architecture diagram above the fold, a how-to-run block, and the 2 or 3 metrics that prove it works (rows/day, end-to-end latency, cost). The README carries more hiring weight than the code.
  3. 03

    Show the architecture, not just the code

    Every strong repo on this page leads with a diagram. Reviewers evaluate boxes and arrows before they open a single file, and the diagram is what they will ask you to defend in a system design round.
  4. 04

    Rehearse the defense

    Why Kafka instead of a queue? Why is this load idempotent? What happens when the API changes a field? Building the projects gives you the answers; saying them out loud under time pressure is a separate skill. Rehearse with a friend or in a mock interview before the real loop.

Presenting the work

Resume screens and recruiter calls reward the visible artifacts: the README, the diagram, a live dashboard. The loop itself rewards the understanding you built while making them, and every extension you add becomes material for a future system design answer.

For repo structure, README templates, and the order reviewers scan things in, the data engineer portfolio guide covers the presentation side in the same depth as this page covers the projects.

Prepare for the interview
01 / Open invite
02min.

Know projects the way the interviewer who asks it knows it.

a projects query, the same shape a screen would give you.
The diff against expected. Where ties broke. What you missed.
sandbox
1source → bronze → silver → gold
2 ingest : CDC + Kafka
3 transform : dbt + Airflow
4 serve : Snowflake
5
Execute your solution0.4s avg.
DatabricksInterview question
Solve a projects problem

Data engineering projects FAQ

What projects should a data engineer build for a portfolio?+
Build 2 to 3 finished projects that together cover batch orchestration, warehouse or lakehouse modeling, and 1 differentiator (streaming, CDC, data contracts, or lineage). The 15 projects on this page are ordered so that following any 3 in sequence produces that coverage. 1 extended, documented, tested pipeline carries more weight with reviewers than 5 cloned tutorials.
Which data engineering project should I start with?+
If you know SQL but have never shipped a pipeline, start with Jaffle Shop on DuckDB (project 1): it takes an afternoon, costs nothing, and teaches the dbt workflow that appears in 9% of verified DE interview rounds. If you already write software, skip to the Airflow batch pipeline (project 4), which adds orchestration, object storage, and warehouse loading in one readable repo.
Do data engineering projects need to run in the cloud?+
No. 11 of the 15 projects here run entirely on a laptop with Docker, including a full Delta Lake lakehouse and a 3-billion-row warehouse. Hiring managers evaluate architecture decisions, code quality, and your ability to defend trade-offs, none of which require a cloud bill. Add 1 cloud project (the Zoomcamp capstone on GCP's free tier, or a Data on EKS blueprint) when the roles you target name a specific provider.
Are GitHub projects enough to get a data engineering job?+
They get you past the resume screen, which is where candidates without professional DE experience usually fail. The interview loop itself still evaluates SQL, Python, and system design separately, so pair the portfolio with interview practice. A strong project also helps inside the loop: in system design rounds you can walk through an architecture you have built and operated.
Should I put forked tutorial projects on my resume?+
Only after you extend them. Interviewers recognize the popular templates (several appear on this page because they are canonical), and an unmodified clone works against you. Change the data source, add quality gates or a data contract, write the incident notes for something that broke, and record what you would do differently. The extension gives you something to say when they ask what you changed.
What about machine learning projects for data engineers?+
Useful as a 4th project, not a substitute for the core 3. If the roles you target mention ML platforms or feature stores, the MLOps Zoomcamp (from the same DataTalksClub team as project 5) covers experiment tracking, orchestrated training pipelines, and drift monitoring on the NYC taxi dataset, and its capstone stands alone as a portfolio piece.
02 / Why practice

Pair the portfolio with interview practice

  1. 01

    Reading a solution is not the same as writing one

    Every engineer who has frozen on a query they had read a dozen times knows the gap. The only preparation that closes it is producing the answer yourself, under time, before the interview does it for you

  2. 02

    76% of hiring managers reject on the coding task, not the resume

    From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice

  3. 03

    System design comes down to the calls you defend out loud

    Ingestion, batch vs streaming, the bronze/silver/gold layers, idempotency, backfill and replay. Sketching the pipeline and naming the failure modes is the signal, not the boxes

Start practicing

Related career guides

Data Engineer Portfolio Guide→

Repo structure, README templates, and what hiring managers scan for in the first minute

Data Engineering Roadmap→

Which skills to build in what order, from first pipeline to senior scope

How to Become a Data Engineer→

The full career transition path these projects slot into

Data Engineer Resume Examples→

How to write the resume bullets these projects earn you

Data Engineering Tools Guide→

Every tool in these stacks, ranked by how often it comes up in interviews

Data Engineer Interview Questions→

The questions these projects prepare you to answer, by domain and round

Interview Prep

Data Engineer Interview Prep Guide→

50+ guides covering every round, company, and role

Interview Questions
SQLPythonData ModelingData PipelinePySparkAI Coding
Practice Problems
SQLPythonData ModelingData PipelinePySparkAI Coding
Guides & Tools
Data Engineer ResumeData Engineer RoadmapTop 100 QuestionsDE vs Data AnalystKafka vs KinesisData Mesh
By Company
NetflixAirbnbLyftUberTeslaSnowflake