Safe to Run Again

The title on your offer letter says data engineer

The company that wrote it could not tell you what that means. Stretched to mean everything, the title means nothing. Nobody knows what a data engineer is anymore, except data engineers.

Somewhere right now another data engineer is doing exactly this, inside a system that looks nothing like yours. A scheduler that predates his employment. 400 stored procedures nobody understands anymore. A platform moving a few billion events a day, or a company where he is the entire data operation. The work looks nothing alike. The instinct is identical, and you would recognize it in him within a minute, like an accent from home.

That instinct is the only real definition this profession has ever had. And it is written down nowhere: not in the job postings, not in the interviews, not in anything the industry has ever said about what you are.

The industry cannot see the instinct, only tool lists

I have read thousands of data engineer job postings, closely, the way you read a query plan when something is wrong. Under this one title there are 5 different jobs: a DBA keeping the lights on, an analyst building dashboards, a transformation engineer living in SQL models, an ML babysitter feeding features to someone else’s model, and an infrastructure engineer who is a backend developer in everything but name. Most postings blend 2 or 3 of them, plus whatever the market is currently infatuated with.

There is a word for what happened here: title laundering. Employers discovered that any unstaffed data problem could be laundered through the words “data engineer” and come out the other side looking like a single, hireable role. 5 headcount gaps, 1 requisition. A relabeled DBA here, a rebranded report writer there, a “modern data platform” wish list stapled on top. The title takes everything and objects to nothing, because a title cannot object. Only the people holding it can.

Data engineers did not stretch this word. The stretching was done by people who never ran a backfill: job descriptions written without a single technical review, demanding 5 years of experience with tools that have existed for 3; org charts that needed one box to absorb every data responsibility nobody else wanted; the relabeling reflex that renames roles to chase whatever word is fashionable this cycle. A job title is supposed to be a contract. It tells the world what the work is, so the work can be respected, priced, interviewed for, and taught. The industry broke the contract, kept the word, and called it flexibility.

If you want proof, watch how they hire

You sit down for a data engineering interview and someone asks you to invert a binary tree. The job is making a pipeline safe to run twice. You get asked to sort an array on a whiteboard, from memory. The job is knowing the file landed twice because the vendor’s SFTP retried, and that your merge logic is the only thing between that retry and a finance report that is quietly wrong. Nobody has ever deployed a linked list at 3 in the morning. The interview is borrowed from another profession because institutions test what is standardized, and nothing about this role ever was. They cannot test the true thing. They never agreed on what it is.

All of it lands somewhere. It lands on you. It has a schedule, and its schedule is night.

Because the title means nothing, anything can be assigned to it. The roadmap you wrote in January is a memory by March, buried under requests that were each going to take 5 minutes. The dashboards you shipped are open in nobody’s browser. The work you are proudest of is invisible by design: pipelines that run correctly are silent, and silence earns nothing. You become visible exactly once, when something breaks, and the thing that breaks is almost never yours. An engineer 3 teams away renames a column on a Friday. You find out on Monday, from an executive, in a tone. The producer who broke the data will never be paged for it. You will.

Every data engineer I respect carries versions of the same scars. The 3am page for a job that ran clean for 2 years. The backfill that changed last quarter’s numbers while everyone watched, and the meeting afterward where explaining why the old numbers were wrong somehow made you the person who wronged them. And the one we all fear: the job that ran green for months while the data inside was garbage. Exit code 0. Dashboards plausible. 3 months of a metric quietly poisoned, found by accident, fixed in shame. Every scar traces to the same disease. Nobody agreed what this job is, so it became all jobs, including the ones nobody could do safely.

The same disease sawed off the ladder

Ask around and you will hear, with a straight face, that data engineering is not an entry-level job. Then read the entry-level postings, the few that exist, and find them demanding end-to-end ownership of ingestion, modeling, orchestration, cost, and governance, in a stack the company cannot describe. Both statements come from the same industry, often the same company, sometimes the same paragraph. A profession that cannot define itself cannot train anyone into itself. It can only poach people who already became the thing somewhere else, and it could not tell you how they managed it.

Every craft that survived built a ladder: named stages, known skills, a visible path from day 1 to the person others call at 3am. Ours never got built. What got sold instead was certificates, receipts for courses stacked where the rungs should be, and anyone who has interviewed a certified candidate who could not explain their own pipeline knows what those receipts are worth. The people who pay are the ones trying to get in, told to enter and that there is no entrance, in the same breath. The people already inside pay too, alone on call, because nobody was ever grown to stand beside them.

Here is what the industry missed

There has always been a real definition of this job. It just never got written on anything official. The job postings define nothing at all. The definition lives in the heads of the engineers who run these systems, and it gets enforced where it always has been: code review, incident channels, the quiet judgment of whoever operates what you built.

The real definition is not a tool list. The tools are real engineering, and mastering them is part of the job, but they turn over with every era: clusters, then cloud warehouses, now AI. There will be another era. A definition that turns over with the tools is not a definition. That is how the title got stretched in the first place.

The real definition is a set of instincts.

It starts with the difference between “the job succeeded” and “the job is safe to run again.” Everyone begins by celebrating the first. The craft is designing for the second, because the job will be rerun, by a retry, by a backfill, by a tired human at 2am who needs the rerun to be boring.

It is knowing that exactly-once delivery is a bedtime story vendors tell, and that idempotency is what protects you when the story ends.

It is respect for data that arrives late, out of order, or changed, from sources you do not control and never will. The reflex that asks “what happens when this is wrong” before anyone asks “how fast can this be done.”

It is modeling data so the person after you can reason about it without archaeology. Treating cost as an engineering constraint rather than an accounting complaint, because a pipeline the business cannot afford is a failed pipeline that happens to work.

It is choosing boring technology on purpose, without shame, because you have seen what excitement costs at scale and at night.

And it is the strange, specific pride of platform work done well: building the thing so solidly that people use it every day without knowing your name. The best data engineers I know are proudest of systems nobody notices. That pride is the craft.

If you felt recognized just now, that recognition is the point. Nobody taught us any of this in a course. It accreted, incident by incident, in production. It is the tacit standard we already hold each other to, more consistent across companies and continents than any job description has ever been. We converged on a definition years ago. The market never did.

None of it is syntax, and there is a reason. Tools have always written some of our code, and every wave writes more. The next will write more still. Each wave produces a cottage industry of people announcing the role is dying, the same announcement that greeted every wave before, delivered each time with total confidence and a terrible memory.

Here is what actually happens, every time. The parts of the job that were only syntax get automated, and the people whose jobs were only syntax get hurt, and that part is real. What remains after the wave is the job, concentrated. What to build and what to refuse to build. What a number is allowed to mean. What the blast radius is when the upstream lies, who gets woken, whether the rerun is safe. Judgment was never the part any wave could write. Syntax was never the profession. The waves keep proving it, the title launderers keep not noticing, and the gap between what the industry hires for and what the work requires keeps widening.

What we are going to do about it

We are done waiting for the industry to define this profession. Employers define the role as whatever gap needs plugging this quarter. Vendors define it as whatever their platform sells. Each of them holds the pen for a while, and each writes down something that is not the job. The one group that never got to hold the pen is the group that actually knows: the people who do the work.

So the definition moves. From now on, it lives with the people who do the work.

That means a standard written and maintained in public, versioned like everything else we maintain, revised the way our field actually revises things: out loud, with arguments, by people with scars. And it means proof, because a definition without proof is just another job description. This field trusts one kind of proof: something built, against real data, under realistic failure, where anyone can inspect what happened. A resume can claim it. A badge can gesture at it. Proof is a thing that ran, and reran, and was safe both times.

That is what we are building. None of it depends on any platform, including ours. The declaration stands on its own: the title got stretched by people who do not do the work, and the definition belongs to the people who do.

One more thing, about who “we” is, because it would be easy to build this wall in the wrong place.

The line does not run between year 1 and year 20, or streaming and batch, or between the engineer with the billion events and the engineer with the 400 stored procedures, who are, and I mean this, both doing the real job. Scale was never the badge. Some of the best guardians of this craft are a year in and already ask “is this safe to rerun” before anyone taught them to. Some people with impressive titles have never once asked it.

The line runs between people who want this title to mean something and people who profit from it meaning nothing.

You know which side you are on. You knew before you got here. If you have held your breath during a rerun, if you have smirked at a vendor’s exactly-once slide, if you have fixed a number at night for someone who will never know it was broken, you are already one of us, and none of it required anyone’s permission. And if you are just getting started, here is the version the job ads will not give you: this work is hard, the hard parts are the good parts, and now at least one description of the destination is true.

The standard stays public. The ladder stays visible. The people who hold the title hold the definition.

It’s late, and somewhere a job just went green