Open Source ETL Tools Compared

Airbyte is a data movement platform with a catalogue of 700+ connectors, offered self-hosted or as a paid cloud service. Its licensing is the detail to read first. The Airbyte licences page says its connectors and public repositories, except the Airbyte Protocol, are under the Elastic License 2.0 and that only the protocol is MIT. The 2 uses it names as restricted are hosting Airbyte yourself to sell it as an ELT or ETL tool and selling a product that directly exposes its UI or API. Running it for your own company's pipelines is neither. It fits teams that want a large prebuilt connector catalogue and no connector code.

dlt is a Python library under Apache 2.0 (6.1k stars), not a platform. You define a source in Python, create a pipeline with a destination such as DuckDB or Snowflake, and call pipeline.run(source). It infers the schema and normalizes nested JSON into tables. It also supports incremental loading and schema evolution, and it loads to 20+ destinations. Because it is a library it runs wherever Python runs, such as an Airflow task or a container, and you own the scheduling and infrastructure. It fits teams that write Python and have APIs no connector catalogue covers.

Meltano (MIT, 2.7k stars) is built around the Singer specification. A Singer tap writes JSON lines to stdout in 3 message types: SCHEMA and RECORD describe and carry the data, and STATE records progress. A target loads them, and the STATE message is the bookmark the next run reads to resume where the last one stopped. Meltano wraps taps and targets in a declarative, code-first project and tracks that state for you. It runs dbt as a transformer and supports Airflow as its orchestrator. It fits teams that want a pipeline defined as files in Git and an existing tap for their source.

Apache NiFi (Apache 2.0, 6.3k stars) moves data between systems as flows you design in a browser. Its documentation highlights data provenance, which records the full lineage of every piece of data, a choice between guaranteed and loss-tolerant delivery, and back pressure that slows producers when consumers fall behind. Those are operational guarantees, which is why it suits routing across many protocols and audit requirements more than warehouse-centred ELT.

Apache Hop is a visual platform for pipelines and workflows with 250+ transforms. A run configuration picks where the same pipeline runs: its native engine, a Hop Server, Spark or Databricks, or Flink and Google Dataflow through Apache Beam. It suits teams that want a canvas, transformation inside the tool and a portable pipeline.

Apache SeaTunnel is a top-level Apache project for batch and streaming work and for change data capture, with about 200 connectors. A job definition runs on its own Zeta engine, or on Flink or Spark, so it suits teams that already operate one of those engines.

Talend Studio shows the risk of visual ETL tools. Qlik retired the open source Talend Studio on 31 January 2024 and no longer hosts or updates it.

Debezium (Apache 2.0) turns a database's row-level writes into a stream of change events. It lists 18 source connectors, 8 of them stable, among them CockroachDB, Db2, MariaDB, MongoDB, MySQL, Oracle, PostgreSQL and SQL Server. Each event carries the operation and the row before and after the change, plus source metadata.

Debezium normally runs inside Kafka Connect and writes to Kafka topics, with offsets stored in Kafka so it resumes in commit order after downtime. Debezium Server runs the same connectors without Kafka. It is the tool for propagating row-level changes in near real time, including deletes that a nightly extract misses, and it feeds a warehouse only through something that consumes the stream. Interviewers ask about this pattern often.

dbt handles the T of ELT. You write SQL select statements and it builds tables and views in the warehouse in dependency order, and its documentation says it works alongside ingestion tools rather than replacing them. Its licence differs between the Python-based v1 and the newer Rust-based releases, so read the licensing page for the version you would run.

Airflow (Apache 2.0, 47.4k stars) and Dagster (Apache 2.0, 16.4k stars) define pipelines in Python and run other tools in order, retrying them on failure. Airflow models work as tasks in a DAG, and Dagster models it as assets, the tables and files the pipeline produces. Neither moves data by itself at scale; they call the loaders and dbt. The dbt vs Airflow guide explains how the 2 roles divide.

Open source is not one thing. Apache 2.0 and MIT licences place few limits on how you run a tool. The Elastic License 2.0 allows you to read and run the code but restricts offering it as a service, which is why Airbyte is better called source-available. Check the licence of the exact component you plan to deploy, because a platform and its connectors can carry different terms, and the protocol can differ again.

Governance decides how long a tool lives. Airflow, NiFi, Hop and SeaTunnel are Apache Software Foundation projects, and Debezium is hosted by the Commonhaus Foundation. Talend Open Studio sat under one vendor, and its retirement on 31 January 2024 left users with software nobody updates. Before adopting a tool, look at who commits to it, how often it releases and whether a company could close it.

The licence is free and the operation is not. A self-hosted tool leaves you with upgrades, secrets, scaling and on-call monitoring. Connectors break when a source changes its API. A library such as dlt leaves the scheduler and the compute to you, and a continuous service such as NiFi or Debezium needs a supervisor and alerts of its own.

Make every load idempotent, so a retry cannot double rows. Store incremental state outside the worker, since a lost bookmark forces a full reload. Alert on freshness and row counts at the destination, because a loader that exits 0 can still load nothing.

Interviewers rarely ask you to rank tools. They give a pipeline problem and listen for whether you can name what each stage needs: how a source is read incrementally, where state is kept, what happens to a late or duplicated record, and how a backfill runs without breaking the current load.

To practice that reasoning, work through the ETL interview questions and the data pipeline interview questions, then design full pipelines in the data pipeline practice problems. The data engineer interview prep guide lays out the whole loop, and data pipeline architecture shows how the pieces above fit together.