An hourly Spark job writes into a daily-partitioned plain Snowflake table with INSERT INTO

A medium Pipeline Design interview practice problem on DataDriven. Write and execute real pipeline design code with instant grading.

Domain

Pipeline Design

Difficulty

medium

Problem

An hourly Spark job writes into a daily-partitioned plain Snowflake table with INSERT INTO. The section just named this exact mismatch: when run cadence is finer than partition grain, each retry appends on top and a partial-failure retry obliterates the day's accumulated work. Apply both halves of the section's recipe: an ACID destination and an idempotent intermediate step so the chain is safe end to end.

Practice This Problem

Solve this Pipeline Design problem with real code execution. DataDriven runs your solution and grades it automatically.