Pipeline Testing

Concepts: paCiCd

What They Want to Hear 'Four levels: static analysis (seconds, lint + type check), unit tests (minutes, test individual transforms), integration tests (30 min to 1 hour, end-to-end with real data), data diff (hours, compare production outputs). The most important single test is schema compatibility: assert that the output schema has not changed unexpectedly. Schema breaks are the #1 cause of production data incidents.'