State Backend Architecture

Concepts: paMicroBatchVsTrue

What They Want to Hear 'For large state, I use RocksDB as the state backend. RocksDB stores state on local disk instead of the JVM heap, so it can handle state larger than memory. Checkpoints write state snapshots to a distributed filesystem (S3 or HDFS). For 500GB of state, I use incremental checkpoints: each checkpoint writes only the changed data since the last checkpoint, reducing checkpoint duration from minutes to seconds. The critical tuning parameters are checkpoint interval (balance between recovery time and checkpoint overhead) and max concurrent checkpoints (never more than 1 to avoid checkpoint storms).' This is the answer that shows you have operated large-state Flink jobs.