Data Engineer Skills

The site tracks which tools data engineering postings name. Of 36,543 postings that name at least 1 tool, 47.8% name AWS and 36.8% name CI/CD. Kubernetes is named in 32.6% and Azure in 29.4%. Cloud and deployment skills lead the table, and no data-specific tool gets past 12.8%.

That ordering surprises people who expect Spark and Airflow at the top. It happens because a cloud or a container platform is a requirement of almost every platform job, while a processing engine is a requirement only of the jobs that run that engine. A posting for a Snowflake shop never names Spark, and a posting for a Spark shop never names dbt. Demand for the data tools is split across several tools doing overlapping work, and demand for the cloud is not.

The dataset holds tools and not languages, so SQL and Python have no share in it. Their demand shows up indirectly. dbt models are SQL and Airflow pipelines are Python files, while Spark runs both. Treat SQL and Python as the entry fee and the tools in the chart as the specialization that decides which jobs fit you.

SQL is the one skill with no substitute. Every warehouse in the data speaks it, from Snowflake at 9.9% and Databricks at 10.1% to BigQuery at 5.1% and Redshift at 3.0%, and PostgreSQL is the most named database at 14.9%. dbt, which builds warehouse tables from SELECT statements, appears in 5.2%.

The level that matters goes beyond SELECT and JOIN. Window functions for ranking and running totals and aggregation with correct NULL handling are the base. Deduplication with ROW_NUMBER() and joins that don't fan out rows are what separate a data engineer from someone who can query. Reading a query plan to see why a join is slow comes next.

Interviews test this directly, in a timed round with real tables. The SQL interview questions cover what that round asks, and the SQL practice problems let you write the queries against data and see whether they pass.

Python is how a data engineer does what SQL cannot: call an API, parse a file with an irregular layout, write a custom check, or define a workflow. Airflow, which appears in 7.5% of postings, defines each pipeline as a Python file, and PySpark is the usual way to drive Spark from Python.

The skills that carry over are plain ones: dictionaries and sets for lookups, generators for data that doesn't fit in memory, comprehension over loops, careful error handling around network calls and tests that run without the network. Pandas matters for small data, and the habit of asking how much memory a step needs matters more.

Python rounds in data engineering interviews ask for transformation logic, not algorithm puzzles: group, deduplicate, merge, parse. The Python interview questions show the shape of them, and the Python practice problems are written for that style.

Data modeling is the design of tables: what 1 row means, which keys join them and how history is kept. It appears in no tool column, so it's easy to miss in a count of postings, yet it decides whether a warehouse stays usable at 100 tables.

Start with grain, the statement of what 1 row of a table represents. A fact table at order-line grain can answer a question about products, while the same table at order grain cannot. Then learn the standard patterns: fact and dimension tables in a star schema with surrogate keys, and slowly changing dimensions, where a type 2 dimension keeps 1 row per version of a customer with valid-from and valid-to dates.

Modeling is also its own interview round, where you draw a schema for a business and defend it. The data modeling interview questions list what gets asked, and the data modeling practice problems let you design models before you have to do it live.

Learn 1 cloud well enough to build a pipeline on it without a tutorial open. AWS leads with 47.8% of postings, Azure follows at 29.4% and GCP at 27.0%. AWS has 1.6 times as many postings as Azure, so it's the default when you have no preference.

What to learn is the same on each. Object storage is the landing zone, with a managed warehouse or lakehouse engine on top of it. Identity and permissions and the networking that lets a job reach a database come with them. The skill that transfers is knowing which service plays which role, and the names are the part that differs.

Choose by the employers you want. A data engineer who targets banks and large enterprises meets Azure often, one who targets startups meets AWS and GCP, and a job hunt is faster with the platform your target companies already run.

The deployment tools appear in more postings than any data tool: CI/CD in 36.8% and Kubernetes in 32.6%. Docker is in 22.6% and Terraform in 18.1%. The postings include roles with infrastructure duties, so read these as what employers want beside data skills and not as skills only data engineers use.

The working minimum is small. Package a job in a Docker image so it runs the same on a laptop and in production, keep the pipeline code in version control with a CI step that runs its tests, and describe cloud resources in Terraform so that a bucket or a role is reviewed like code. Kubernetes matters when your employer runs Airflow or Spark on it, and you can defer it until then.

3 skills decide how far an engineer goes and none of them is a tool. The first is data quality: knowing a table is wrong before an analyst finds it, whether it arrived late or short, which data observability formalizes with checks on freshness and volume, and on schema. The second is cost awareness, since a badly written query on a cloud warehouse is billed by what it scans. The third is written communication of tradeoffs to people who do not read code.

These skills are tested in the system design and behavioral rounds, where the question is rarely which tool you would pick and nearly always why and what breaks first.

A list of skills is only useful with an order and a deadline. The data engineer roadmap turns it into 6 stages in sequence, and timed practice on real SQL and pipeline problems tells you which group of skills is weakest before an interviewer does.