Data Engineering Glossary

These terms describe how data leaves a source system and arrives somewhere it can be used. Most of them are choices along 2 axes: how often data moves (in bounded batches or as a continuous stream) and where it's reshaped (before loading or after).

Data pipeline. A series of steps that moves data from its sources to a destination and changes it on the way, run on a schedule or continuously.

Source system. Where a piece of data originates, such as an application database, an API, an event stream or a file drop. A pipeline reads from sources and never writes to them.

Sink. The destination a pipeline writes to, such as a warehouse table, an object storage path or a topic.

Ingestion. The step that copies data from a source into the platform's own storage, usually with little change to its shape.

ETL. Extract, transform, load. Data is reshaped in a processing layer before it reaches the warehouse, so the warehouse only holds the cleaned result.

ELT. Extract, load, transform. Raw data is loaded first and reshaped inside the warehouse with SQL, which keeps the raw copy available for reprocessing.

Batch processing. Processing a bounded set of data in 1 run, such as all orders from yesterday. The job has a start and an end, and it reports success or failure.

Stream processing. Processing events continuously as they arrive, on data that has no end. State and event ordering get hard, and late events make it worse.

Micro-batch. Batches so small and frequent that they behave like a stream. The processing model is still batch, with a latency of seconds to minutes.

Change data capture (CDC). Reading a database's change log to capture every row change instead of repeatedly querying the table. It carries deletes that a query on the table cannot see.

Full load. A run that copies the whole source table every time. It is simple and always correct, and it gets slow as the table grows.

Incremental load. A run that moves only the rows that are new or changed since the last run, tracked by a high-water mark such as the latest updated_at already loaded.

Backfill. Re-running a pipeline over past periods to fill a gap or correct history after a bug or a late source. A pipeline that can't backfill safely can't be repaired.

A pipeline that works once is easy. These terms cover what makes it keep working: running tasks in order, and staying correct when a task is retried or data shows up late.

Orchestration. Scheduling and coordinating the tasks of a pipeline: running them in dependency order and retrying failures, then alerting a person when retries run out.

DAG. A directed acyclic graph: tasks as nodes and dependencies as one-way edges, with no cycles, so there's always an order in which every task can run. Airflow and Dagster both describe a workflow this way.

Idempotency. Running a task more than once with the same input leaves the same result as running it once. It's what makes retries and backfills safe. Idempotent pipelines get their own guide.

At-least-once delivery. Every message is delivered 1 or more times. Nothing is lost and duplicates are possible, so the consumer must tolerate repeats.

At-most-once delivery. Every message is delivered 0 or 1 times. Nothing is duplicated and a failure can lose data.

Exactly-once. The effect of each message lands once. It is usually built from at-least-once delivery plus an idempotent or transactional write, not from a network that never repeats.

Dead letter queue. A holding queue for messages that failed processing, so a bad record doesn't block the stream and can be inspected and replayed later.

Event time and processing time. Event time is when something happened at the source. Processing time is when the pipeline saw it. The gap between them is why streaming results need windows and watermarks.

Late-arriving data. Records that reach the pipeline after the period they belong to has already been processed, such as a mobile event sent when the phone comes back online.

Watermark. A stream processor's estimate of how far event time has progressed. A window closes once the watermark passes its end, so the watermark sets how long late events are waited for.

Backpressure. A signal from a slow consumer telling upstream stages to slow down, so a queue doesn't grow until memory runs out.

Data contract. An agreed and enforced description of what a producer delivers: its schema, the meaning of its fields, its freshness and its owner. A breaking change is then a reviewed event, not a surprise.

Schema evolution. Changing a table's or event's columns over time. A change is backward compatible when readers built for the old schema keep working, for instance when a new column is optional.

SLA and SLO. A service level agreement is the promise made to consumers, such as a table fresh by 07:30. A service level objective is the stricter internal target set to keep that promise.

Dimensional modeling gives a warehouse a vocabulary that analysts and engineers share. Every term here hangs off the grain, because the grain decides what each row means and which other columns are allowed.

Grain. What a single fact table row represents. Kimball's rule is that the grain is declared first, before dimensions and facts are chosen, and that it then binds the design. The Kimball Group's grain technique recommends starting at the lowest level at which the business process captures data.

Fact table. A table of measurements at 1 declared grain, such as 1 row per order line with quantity and price, plus keys pointing to its dimensions.

Dimension table. A table of descriptive context that facts are sliced by, such as a customer dimension or a date dimension, with 1 row per member.

Star schema. A fact table in the middle with 1 denormalized dimension table around it per dimension. Queries join the fact to the dimensions they need and nothing deeper.

Snowflake schema. A star schema whose dimensions are normalized into several linked tables. It saves some storage and costs extra joins.

Surrogate key. A meaningless integer or hash the warehouse assigns to a dimension row, used in place of the source's natural key so history and source changes can't break joins.

Natural key. An identifier that exists in the source, such as an order number or an email address. The source is free to reuse or change it, including its format.

Slowly changing dimension (SCD). A dimension whose attributes change over time, with a rule per attribute for what to do about the old value. The types are listed in the slowly changing dimension types ledger, and slowly changing dimensions has the SQL.

Conformed dimension. A dimension shared by several fact tables with the same keys and meaning, so their measures can be compared on the same attributes.

Degenerate dimension. A dimension key that lives in the fact table with no dimension table of its own, such as an order number.

Normalization. Splitting data into tables so each fact is stored once. It protects transactional systems from update anomalies.

Denormalization. Copying descriptive values into wider tables so analytical queries join less. It trades storage and update effort for read speed.

Data vault. A modeling method that separates business keys (hubs), the relationships between them (links) and their changing attributes (satellites), built for auditable history from many sources.

Interviewers ask you to declare the grain out loud before drawing a table. Practice that habit on the data modeling interview questions and with the data modeling practice problems.

Interviews test these terms by asking you to write the query or explain the slow job. The SQL terms are about expressing a result, and the Spark terms are about why a correct result takes an hour.

Window function. A function computed over a set of rows related to the current row, defined by OVER (PARTITION BY ... ORDER BY ...), that returns a value on every row instead of collapsing them like GROUP BY.

CTE. A common table expression: a named result defined with WITH that later parts of the same query can read, used to break a long query into readable steps.

Upsert. Update the row if its key exists, insert it if not. SQL spells it MERGE in most warehouses and INSERT ... ON CONFLICT in PostgreSQL. A keyed upsert is a common way to make a load idempotent.

Cardinality. In modeling, how many rows on each side of a relationship match, from 1 to 1 up to many to many. In a query engine, the number of distinct values in a column.

Shuffle. Moving data between machines to bring rows with the same key together; a join or a GROUP BY requires it. It's the most expensive step in most distributed jobs.

Data skew. An uneven spread of rows across keys, so 1 task receives most of the data and the whole job waits for it.

Broadcast join. Sending a small table to every worker so a large table joins locally without a shuffle.

Predicate pushdown. Applying a filter at the storage read, so data that can't match is never loaded.

Quality terms are about trust: whether data is right, whether anyone would know if it were not, and who is responsible for it.

Data quality. Whether data fits its use: complete, unique, valid, consistent and on time. It is tested with rules such as order_id is never null.

Data freshness. The time since a table last received new data, judged against how stale that table is allowed to get.

Data observability. Watching the tables a pipeline produces, through freshness, volume, schema, lineage and values, so a problem is found by a check before it is found on a dashboard. What is data observability goes through each check.

Data lineage. The record of which jobs read and wrote which datasets. Upstream it locates a root cause, and downstream it lists what a change will break. OpenLineage defines a job as a process that consumes or produces datasets and a run as one occurrence of it.

Metadata. Data about data: schemas, owners, descriptions, row counts and load times. Most observability and cataloging is built on it.

Data catalog. An inventory of an organization's datasets that lists each one's owner and description and traces its lineage, so people find and trust data without asking in chat.

Data governance. The policies and roles that decide who may use which data and how, covering access, retention, privacy and ownership.

Data mesh. An organizing approach in which the teams that know a domain own its data as a product, on a shared platform, instead of one central team owning every pipeline.

Interviewers rarely ask for a definition. They use a term inside a scenario and expect you to know what it commits you to: "make this load safe to rerun" is idempotency, "how do you handle events that arrive after the window closed" is watermarks and late data, "what is 1 row in this table" is grain. Knowing the word lets you skip to the decision.

A definition worth giving has 3 parts: what it is, the problem it solves and the cost it brings. Idempotency makes retries safe and costs you a keyed write. A star schema makes queries simple and costs you redundancy. Practice saying those 3 parts for every term in this glossary, then apply them to a design in the data pipeline interview questions, and build the pipelines themselves with the data pipeline practice problems.

For the full loop, from SQL to system design, start at the data engineer interview prep hub. If slowly changing dimensions are the term you want to drill, the SCD interview questions go deeper than a list of types.