Phase 6 · Messaging & Data Pipelines
Queues, Kafka, async failure modes and stream processing.
- Message Queues & Kafka Internals: Queues vs pub/sub vs logs, Kafka partitions, replication, ISR and acks, consumer groups, offsets, ordering and exactly-once — plus when RabbitMQ or SQS is the better tool.
- Async Processing Failure Modes: The production incidents of queue-based systems: slow APIs, 10× spikes, lag, crashes mid-message, duplicates, poison pills, ordering under retries, hot partitions and backpressure. For each one: symptoms, root cause, fix.
- Data Pipelines: Batch, Streaming & CDC: ETL vs ELT, batch vs stream, lambda vs kappa, windows, event time and watermarks, exactly-once checkpoints, CDC with Debezium, lakes vs warehouses, real-time dashboards and reconciliation.