Async Processing Failure Modes
The production incidents of queue-based systems: slow APIs, 10× spikes, lag, crashes mid-message, duplicates, poison pills, ordering under retries, hot partitions and backpressure. For each one: symptoms, root cause, fix.
About 105 minutes · 15 exam questions · 9 interview questions · 2 code labs
What you will learn
- A mental model for async incidents: rates, lag and Little's law
- Scenario: the API takes 10 seconds → 202 Accepted, polling and webhooks
- Scenario: traffic spikes 10× → queue buffering and load leveling
- Scenario: the consumer is slower than the producer
- Scenario: the queue keeps growing — a triage runbook
- Scenario: the consumer crashes mid-processing
- Scenario: a message is processed twice → idempotent consumers
- Scenario: repeated failures → retry topics, DLQ, poison pills and redrive
- Scenario: ordering is required — keys, per-key serialization and sequence numbers
- Scenario: one consumer is a bottleneck / hot Kafka partition
- Scenario: downstream is overwhelmed → backpressure end to end
Previous: Message Queues & Kafka Internals · Next: Data Pipelines: Batch, Streaming & CDC