Build vs Buy at Each Layer
Concepts covered: ELT vs ETL
Every layer of a pipeline can be built in-house or bought as a managed service: the connectors that pull from sources, the storage the curated layer sits in, the transform logic that encodes your definitions, the engine that runs that logic, the scheduler that runs the DAGs, and the tools that serve the output. The decision is made per layer, and the same team reasonably builds 1 of the 6 and buys the other 5. Each decision comes down to 3 weights.
The 3 weights
- Differentiation: A connector that pulls a payment provider's API is identical whoever writes it; the consumer never sees it. A transform that encodes how your company defines an active user is yours alone, and buying it means buying someone else's definition.
- Total cost of ownership: The build is the cheap part. A connector takes 6 engineer-weeks to write and 1 engineer-day a month forever to keep working as the source changes, which is 2.5 more engineer-weeks a year, every year.
- Strategic risk: A bought storage layer holds your data in its format; leaving means a migration measured in months. A bought connector holds nothing; leaving means rewriting 1 pull. Risk is highest where data and logic accumulate.
| Layer | Default | Exception |
|---|---|---|
| Ingest connectors | buy | the source is internal and proprietary, so no vendor sells a connector for it |
| Storage | buy | regulation requires the bytes to stay on hardware you control |
| Transform logic | build | the logic is a commodity, such as currency conversion from a published rate table |
| Transform engine | buy | a query shape the managed engine cannot finish inside the contract's ready-by time |
| Scheduler | buy | schedules depend on signals from internal systems the managed scheduler cannot see |
| Serving | buy | the consumer is a product feature with latency needs no dashboard tool meets |
A worked total cost
The listening data team needs sessions from a third-party podcast host's API. A managed connector costs $1,500 a month. Building takes 1 engineer for 6 weeks, and the API changes often enough that it needs 1 engineer-day a month after that. At a loaded cost of $15,000 a month per engineer, the build is $22,500 up front and $750 a month to maintain. Year 1 costs $31,500 built against $18,000 bought. Year 3 cumulative is $49,500 built against $54,000 bought. The crossover lands in month 30. Because the connector is undifferentiated, the team buys it and spends the 6 weeks on transforms instead. The same arithmetic with a $5,000 a month connector crosses over in month 6 and points the other way, so each layer needs the calculation redone with its own prices.
- Do
- Decide per layer; buying storage and building transforms is the common shape
- Count maintenance in the build cost, in engineer-days per month, forever
- Ask which layer your data and logic would be stuck in if the vendor vanished
- Avoid
- Build a connector to a public API because it looks like a 2-week job
- Buy transform logic; the active-user definition is the product
- Compare year-1 costs only; the crossover is usually in year 2 or 3
A team is choosing where the transform that defines a 'paying customer' lives. Finance and sales disagree on that definition, and support has a third version. A vendor sells a package that computes paying customers from billing data. Which weight decides this, and which way?
- Total cost of ownership: the vendor package is cheaper in year 1, so buy it
- Strategic risk: the vendor could shut down, so build an in-house copy of its package
- Differentiation: the definition is the product, and buying it adopts the vendor's answer to a question 3 teams have not settled
- Total cost of ownership: 3 teams disagreeing means the build will take years, so buy
About this lesson
This section is part of the Pipeline Anatomy: Advanced lesson on DataDriven, the home of data engineering. Lessons cover SQL, Python, Spark, data modeling and pipeline architecture, and each section pairs its explanation with worked examples and practice you run as you go.