Batch Mechanics

Concepts: paBatchProcessing

What They Want to Hear 'I use a high-water mark pattern. The pipeline records the maximum timestamp from the last successful run. On the next run, it only reads rows with a timestamp after that mark. This means we process 50,000 changed rows instead of re-reading 500 million.' That is the core answer. Then add depth: 'I run incremental daily and a full reload weekly as a safety net to catch anything the incremental logic missed.'