The Ingestion Layer

Concepts: paFileIngestion, paApiIngestion, paCdc

Ingestion isn't about choosing between Kafka and file drops. It's about designing a schema-aware, multi-source ingestion framework that handles schema evolution, late-arriving data, and cross-source reconciliation without manual intervention. Schema Registry: The Non-Negotiable The conventional wisdom is 'use Avro with a schema registry.' That's correct but incomplete. The deeper question is: what's your schema evolution policy? Backward compatible changes (adding nullable columns) should auto-propagate. Breaking changes (removing columns, changing types) should fail the pipeline and alert the source team. Forward compatibility - where consumers can read data produced by a newer schema - matters when you have multiple consumer teams upgrading at different speeds. Event Sourcing and Lat