High-Load Data Processing System
A nightly batch pipeline that had outgrown its window, re-architected into an incremental streaming system with correctness checks that alert before customers notice.
- Processing moved from a single nightly window to continuous incremental updates
- Single-day reprocessing without re-running the full pipeline
- Java
- Kafka
- PostgreSQL
- Redis
- Docker