Async Processing Failure Modes

The production incidents of queue-based systems: slow APIs, 10× spikes, lag, crashes mid-message, duplicates, poison pills, ordering under retries, hot partitions and backpressure. For each one: symptoms, root cause, fix.

About 105 minutes · 15 exam questions · 9 interview questions · 2 code labs

What you will learn

  1. A mental model for async incidents: rates, lag and Little's law
  2. Scenario: the API takes 10 seconds → 202 Accepted, polling and webhooks
  3. Scenario: traffic spikes 10× → queue buffering and load leveling
  4. Scenario: the consumer is slower than the producer
  5. Scenario: the queue keeps growing — a triage runbook
  6. Scenario: the consumer crashes mid-processing
  7. Scenario: a message is processed twice → idempotent consumers
  8. Scenario: repeated failures → retry topics, DLQ, poison pills and redrive
  9. Scenario: ordering is required — keys, per-key serialization and sequence numbers
  10. Scenario: one consumer is a bottleneck / hot Kafka partition
  11. Scenario: downstream is overwhelmed → backpressure end to end

Previous: Message Queues & Kafka Internals · Next: Data Pipelines: Batch, Streaming & CDC