Open Source Data Engineering Projects to Contribute To

The open source data engineering projects with the clearest first-contribution paths are dlt, Apache Airflow, dbt (data build tool), Apache Iceberg and Apache Spark, plus the streaming projects Apache Flink and Apache Kafka. This page shows where each one tracks beginner work, how to find good first issues in Apache Airflow, how to contribute to Apache Spark, and what to do before you write code.

Last updated: November 3, 2026Proudly published by: Jeff Wahl10 min read

The 7 projects at a glance

#	Project	Difficulty	Time	Core stack	Cost
01	dlt (data load tool)	Beginner+	1 weekend to a first PR	Python · DuckDB · Docker	$0
02	Apache Airflow	Intermediate	1 to 2 weekends to a first PR	Airflow · Python · Docker	$0
03	dbt (data build tool) core	Intermediate+	2 to 3 weekends to a first PR	dbt · Python · SQL	$0
04	Apache Iceberg	Intermediate+	2 to 3 weekends to a first PR	Iceberg · Java · Spark	$0
05	Apache Spark	Intermediate+	2 to 4 weekends to a first PR	Spark · Scala · Python	$0
06	Apache Flink	Advanced	3 to 4 weekends to a first PR	Flink · Java · Scala	$0
07	Apache Kafka	Advanced	3 to 4 weekends to a first PR	Kafka · Java · Scala	$0

Star counts as each repository's page showed them on 2026-11-03.

Where each project tracks work and how you claim an issue

These projects split into 2 groups. Airflow and dlt track work in GitHub issues with a good first issue label, and Iceberg does too. Spark and Flink track work in JIRA with a starter label for newcomers, and Kafka uses JIRA as well. Claim rules differ more than trackers do. Airflow wants a "working on it" comment and then your pull request. dlt wants a maintainer's confirmation of your approach before you code, and Flink wants a committer to assign the ticket first.

The check you run before opening a pull request differs too. Spark asks for ./dev/run-tests and Flink for mvn clean verify. Iceberg asks for ./gradlew build, and dlt for make lint and make test-common. Run it on an untouched checkout first so you know which failures are yours.

Which project to contribute to first

If your situation is

Pick

Why

You write Python and want the shortest path to a merged pull request

Projects 1 → 2

dlt and Airflow both use GitHub issues with a good first issue label and a Python test loop you can run on a laptop.

You use dbt (data build tool) daily and want to read its source

Projects 3 → 1

Choose the 1.latest branch for Python or main for Rust, then practice the review cycle on dlt.

You read Java or Scala and want a table format or engine

Projects 4 → 5

Iceberg uses GitHub labels, and Spark adds JIRA and a stricter PR title format.

The roles you want name streaming

Projects 5 → 6 → 7

All 3 expect discussion before code, so build the habit of proposing a change on a ticket first.

Python-first projects with a labeled way in

Tier 1 · Projects 1-3

Project 01 · Python ingestion

dlt (data load tool)

Beginner+1 weekend to a first PRLocal & free

You fix a labeled issue in an open source Python library that loads data from APIs and files into destinations such as DuckDB and Postgres, and open the pull request against the devel branch.

The guide is the strictest about process of the 7. New contributors work only on issues labeled help wanted or good first issue. They comment with their intended approach and wait for a maintainer to confirm before writing code. Branches are named {category}/{ticket-id}-description and commits follow Conventional Commits. The loop runs make dev first, then make format and make lint, and ends with make test-common.

The DuckDB destination needs no service, so pytest tests/load -k "duckdb" runs a load test on a laptop. To extend the work into your own project, build a small pipeline with dlt first, hit a rough edge, and file the issue yourself.

Interview signal

A merged change to a library in the ingestion layer shows you can read unfamiliar Python and write a test for a loader, and that you got through a review cycle. The README also says new destinations are unlikely to be merged because of their maintenance cost, so aim at bugs and docs, or at source behavior.

PythonDuckDBDocker

dlt-hub/dlt~6.1k★DataOpen issues labeled good first issue CONTRIBUTING.md

Project 02 · Orchestration

Apache Airflow

Intermediate1 to 2 weekends to a first PRLocal & free

You pick an issue from the good first issue list on Apache Airflow's GitHub contribute page and set up Breeze or a local virtualenv. Then you make the change and open a pull request.

Airflow separates the contribution path by size: the Contributors Quick Start for a first change, a Seasoned Developers guide for a full environment. Issues carry kind:bug or kind:feature labels, and the docs note that unlabeled issues can be just as approachable. You need no issue before a pull request.

Claiming works by comment: write "working on it" and submit the PR. The project doesn't assign issues to non-maintainers, and from March 2026 it does not assign them even when asked. Questions about a first PR go to the #new-contributors Slack channel.

Interview signal

Orchestration is the tool most candidates have used and few have read the source of. A merged fix to a provider or operator, or to the docs, gives you a concrete story about how a DAG is parsed and scheduled, and how it's tested.

AirflowPythonDocker

apache/airflow~47.4k★DataThe repository's good first issues Contributor docs

Project 03 · Transformation

dbt (data build tool) core

Intermediate+2 to 3 weekends to a first PRLocal & free

You contribute to dbt (data build tool), the SQL transformation framework, by choosing between the Python v1 line on the 1.latest branch and the Rust rewrite on main, then following that branch's contributing guide.

The repository is in transition. The README on main describes dbt v2.0 as a ground-up rewrite in Rust and sends readers looking for the Python implementation to 1.latest. The main guide uses cargo build and cargo fmt along with cargo nextest run and cargo clippy. The 1.latest guide needs Python 3.11 or later and runs hatch run setup from the core directory. hatch run unit-tests and hatch run integration-tests handle checks.

Both guides ask for a changelog entry through changie new instead of an edit to CHANGELOG.md. Neither names a good first issue label, so read open issues, and ask in the #dbt-core-development channel of the dbt Community Slack if you get stuck. Read the target branch's guide before you choose an issue.

Interview signal

Knowing how dbt resolves refs into a graph and runs tests on it is a strong answer to analytics engineering questions, and a contribution proves you read the framework instead of only using it. Say which branch you worked on, since the 2 lines differ.

dbtPython

SQL

dbt-labs/dbt-core~14.1k★DataThe repository's open issues CONTRIBUTING.md on 1.latest

Prepare for the interview

01 / Open invite

02min.

Know open source data engineering projects the way the interviewer who asks it knows it.

a open source data engineering projects query, the same shape a screen would give you.

The diff against expected. Where ties broke. What you missed.

sandbox

1source → bronze → silver → gold

2 ingest : CDC + Kafka

3 transform : dbt + Airflow

4 serve : Snowflake

Execute your solution0.4s avg.

MicrosoftInterview question

Solve a open source data engineering projects problem

Engines and table formats: Spark, Flink, Kafka and Iceberg

Tier 2 · Projects 4-7

Project 04 · Table format

Apache Iceberg

Intermediate+2 to 3 weekends to a first PRLocal & free

You take a good first issue in Apache Iceberg, the open table format for analytic tables, build it with Gradle on Java 17 or 21 and open a pull request whose summary starts with an area prefix such as Core: or Spark:.

The loop is ./gradlew build to build and test, ./gradlew spotlessApply to fix style, and ./gradlew revapi to check API compatibility. Add Closes #1234 to the description so the issue closes on merge. New tests use JUnit 5 with AssertJ, and waits use Awaitility instead of Thread.sleep().

Larger ideas go through an Iceberg Improvement Proposal, which uses the Proposal label and a discussion on the dev list. A good extension is to read how a table's snapshots and manifests are written into its metadata for one engine integration, then fix a small bug there.

Interview signal

Table formats are an open design topic in lakehouse interviews. A change to commit logic or scan planning, or to a Spark or Flink integration, shows you understand what a snapshot is and how a reader sees it.

IcebergJavaSparkFlink

apache/iceberg~9.4k★DataIssues labeled good first issue Contribute guide

Project 05 · Batch compute

Apache Spark

Intermediate+2 to 4 weekends to a first PRLocal & free

You find a small, clear change (a documentation fix or an error message, or else a test or a starter JIRA ticket), run ./dev/run-tests on your fork and open a pull request against master titled [SPARK-xxxx][COMPONENT] Title.

Spark tracks work in JIRA and discusses implementation in pull requests, so the first move is to search existing tickets and the mailing list. Trivial changes such as a typo in a scaladoc need no ticket. The project suggests building credibility first by answering questions on user@spark.apache.org and testing releases, then reviewing pull requests and improving documentation.

Before you push, enable the Build and test workflow in your fork, along with Report test results, because CI runs are a shared resource. Core changes, plus anything in SQL or Catalyst, get more scrutiny, so start small. Error messages should say what went wrong and why, and how to fix it.

Interview signal

A merged Spark change, even a small docs fix, is a verifiable line that few candidates have. It gives you a real story about how a query turns into a plan, which is the center of most Spark interview questions.

SparkScalaPythonJava

apache/spark~44.4k★DataJIRA issues labeled starter Contributing to Spark

Project 06 · Stream processing

Apache Flink

Advanced3 to 4 weekends to a first PRLocal & free

You find an unassigned JIRA ticket (newcomers can check the starter label), get a committer to assign it, implement the agreed change and open one pull request for that ticket.

Flink says contributing does not start with opening a pull request. Discuss on a JIRA ticket or the dev list first, and wait, because only committers can assign tickets and unassigned work gets no community review. Large or controversial changes need a [DISCUSS] thread and may need a Flink Improvement Proposal.

Before the pull request, run mvn clean verify plus the end-to-end tests. Then rebase and remove unrelated reformatting. Each JIRA maps to exactly one pull request. Trivial fixes such as typos can go in as a [hotfix] without a ticket.

Interview signal

Following a consensus-first process is itself a working skill. A merged change to a streaming runtime gives you a hard, specific answer about state and checkpoints, or about time handling.

FlinkJavaScala

apache/flink~26.4k★DataUnassigned JIRA issues labeled starter Contribute code

Project 07 · Event streaming

Apache Kafka

Advanced3 to 4 weekends to a first PRLocal & free

You find or file a JIRA ticket, assign it to yourself as the Kafka wiki describes, and open a pull request against trunk titled KAFKA-xxxx: Title.

Changes that alter public APIs or are very visible to users need a Kafka Improvement Proposal, and large changes should be raised on dev@kafka.apache.org before you implement them. The repository's CONTRIBUTING.md points to the contributing page and the wiki guide, and asks you to disclose AI tool use with a commit trailer naming the tool and its version.

Submitting code means you affirm it is your original work and license it under the project's open source license. Start with tests or docs, or with a small bug, and read the consumer group or log code path that a ticket touches before you edit it.

Interview signal

Kafka comes up whenever a design involves events. Having traced one real code path, such as how a consumer group rebalances or how a log segment is read, makes your answers specific.

KafkaJavaScala

apache/kafka~34.1k★DataThe project's JIRA tickets Contributing Code Changes

Good first issues in Apache Airflow

Apache Airflow publishes a contribute page on GitHub that lists open issues carrying the good first issue label, described as chosen for their approachability to first-time contributors. Start there instead of searching the whole tracker, then open the issue and read the comments for a claim from someone else.

The Airflow contributor docs add 2 rules that surprise people. You do not need an issue before opening a pull request, and you shouldn't ask to be assigned: comment "working on it" and submit the change. For a first change, follow the Contributors Quick Start and ask in the #new-contributors Slack channel, which is described as a place where nobody is expected to know much about Airflow yet.

Once the change merges, you can explain how a DAG file is parsed and run from your own diff. The Airflow interview questions cover that vocabulary, and the Airflow DAG guide walks through the objects you will meet in the source.

How to contribute to Apache Spark

Spark's contributing guide recommends a path before large code changes: start by helping users on the mailing list and testing releases, then review pull requests on GitHub and improve documentation, and only after that fix bugs and propose features. The reason is practical. Core changes and SQL or Catalyst changes get the most scrutiny, and a stranger with a track record of useful reviews gets faster feedback.

Work is tracked in JIRA, and the starter label marks a small change suited to new contributors. Search existing tickets before you file one, because duplicates are resolved against the more active ticket. Your pull request title takes the form [SPARK-xxxx][COMPONENT] Title, and tests for a bug fix carry the ticket id in their name, for instance test("SPARK-12345: a short description of the test") in Scala.

The same skills that make a Spark contribution possible are tested in interviews. Practice them with the PySpark interview questions and the PySpark practice problems, and see the PySpark projects for builds that run on your own data.

A first open source contribution, in order

Start with the project's contributing guide, because it names the tracker and the beginner label, and says whether it wants approval before code. Then pick a small issue and follow the claim rule: a comment for Airflow, a ticket discussion for Flink, a maintainer's confirmation for dlt.

Run the build and tests on an untouched checkout, make the change with a test or a doc edit, and keep the diff inside the agreed scope. Then answer review promptly. Push fixes to the same branch and rebase on conflict. Spark notes that stale pull requests may be auto-closed about a week after a committer asks you to close them.

Contributing to dbt (data build tool): Python or Rust

The dbt-core repository changed shape. Its README on main describes dbt v2.0 as a ground-up rewrite in Rust and sends anyone looking for the Python implementation to the 1.latest branch. A contribution to the Python line follows the 1.latest guide. That guide uses hatch run setup and hatch run unit-tests, and also changie new. A contribution to the Rust line follows the main guide. That guide uses cargo build and cargo fmt, and also cargo nextest run.

Neither guide names a good first issue label, so browse the open issues, pick one that is small and ask in the #dbt-core-development channel of the dbt Community Slack when you are stuck. Check the date of any tutorial you follow, since it may describe the other branch.

What maintainers expect from a first change

Maintainers want the change small and inside the agreed scope, with a test shipped alongside it. Follow the format the project asks for, whether that is Spark's pull request title or dlt's Conventional Commits. Iceberg wants an area prefix. And be able to explain every line, since dlt and Flink say so about AI-assisted work, and so do Iceberg and Kafka. Only dlt doesn't also ask you to disclose the tool.

Documentation as a first contribution

Documentation is a legitimate first contribution. Spark keeps release docs as Markdown under docs/ in the main repository and the website in a separate repository, and the Airflow docs list documentation and bug reports as ways to contribute. A docs pull request teaches the review process with the least risk, and you can follow it with a code change to the same area.

Using a contribution in a data engineering interview

A merged pull request is a public artifact an interviewer can open. Describe the problem, what you read in the source to find it, the test you added and what a reviewer asked you to change. That story fits the pipeline design questions and the data pipeline practice problems, where the follow-up is always why the system behaves the way it does.

If you're earlier in the path, the data engineer roadmap places open source work after your first complete pipeline. For builds that are not contributions, the list of data engineering projects has 15 repositories to clone and extend.

Contributing to open source data engineering projects FAQ

How do I find good first issues in Apache Airflow?+

Open the repository's contribute page on GitHub, which lists issues under the heading Good first issues, or filter the issue tracker by the `good first issue` label. The contributor docs say easy bugs are usually marked with that label, and that unlabeled issues can be just as approachable. Comment "working on it" and submit your pull request, because the project doesn't assign issues to non-maintainers.

How do I contribute to Apache Spark?+

Search JIRA and the mailing list for an existing ticket and pick a small change (the `starter` label marks tickets suited to new contributors). Fork the repository and enable the Build and test workflows in your fork. Run `./dev/run-tests`, then open a pull request against `master` titled `[SPARK-xxxx][COMPONENT] Title`. Trivial changes like typos need no ticket. Spark also suggests helping users on the mailing list and testing releases first, then improving docs.

Do I need to write code to contribute?+

No. The Airflow contributor docs count bug reports and feedback as contributions, along with documentation. Spark's guide adds helping users and testing releases, and then reviewing pull requests and improving documentation. Docs and test changes are also the lowest-risk way to learn a project's review process.

Which open source data engineering project is easiest to start with?+

dlt and Apache Airflow are the most direct starts: both are Python projects that track work in GitHub issues under a `good first issue` label, and both document their setup. Spark and Flink track work in JIRA and expect more discussion before code; Kafka does the same.

Are dbt and dbt-core the same thing?+

dbt (data build tool) is the framework, and dbt-core is the repository that holds it. The `main` branch of dbt-core now describes dbt v2.0, a rewrite in Rust, and the Python v1 implementation lives on the `1.latest` branch, so check which branch a guide and an issue refer to.

Why does a maintainer ask me to wait before coding?+

Maintainers of dlt and Flink ask contributors to get approval on an issue before implementing it, because a finished pull request for an unwanted change costs a reviewer time. Comment with your approach and wait for confirmation. Keep the change inside the agreed scope.

Can I use AI tools in a contribution?+

Several of these projects allow it with conditions. dlt and Flink ask you to understand and be able to explain every line, and so do Iceberg and Kafka. Of those, only dlt does not ask you to disclose the tool. Read the project's current guide before you submit.

02 / Why practice

Pair the contribution with interview practice

01
Reading a solution is not the same as writing one
Every engineer who has frozen on a query they had read a dozen times knows the gap. The only preparation that closes it is producing the answer yourself, under time, before the interview does it for you
02
76% of hiring managers reject on the coding task, not the resume
From HackerRank's 2024 Developer Skills Report. Candidates who look strong on paper still fail the live screen if they haven't done timed, executable practice
03
System design comes down to the calls you defend out loud
Ingestion, batch vs streaming, the bronze/silver/gold layers, idempotency, backfill and replay. Sketching the pipeline and naming the failure modes is the signal, not the boxes

Start practicing

Related guides

Data Engineering Projects

15 repositories to build and extend, ordered from a first dbt model to Kubernetes

PySpark Projects

12 Spark builds with real repositories

Airflow DAG Guide

The objects you meet when you read the Airflow source

Data Engineer Roadmap

Which skills to build in what order

The weekly data challenge

Dirty, production-shaped data, scored blind each week.