Incidents III: Async, Dependencies & Resilience

Growing queues, slow and crashing consumers, duplicates, ordering, retries and retry storms, unavailable or slow downstreams, circuit breakers, cascading failures and noisy tenants

About 115 minutes · 15 exam questions · 9 interview questions · 2 code labs

What you will learn

  1. Incident: the queue keeps growing — consumer slower than producer
  2. Incident: consumer crashes during processing (and poison messages)
  3. Incident: duplicate messages — "the customer got charged twice"
  4. Incident: message ordering required — "refund processed before the payment"
  5. Incident: we need a retry mechanism (and the naive one made it worse)
  6. Incident: retry storm — the downstream recovered but we are still down
  7. Incident: a downstream is unavailable
  8. Incident: a downstream is slow (worse than down)
  9. Incident: we need a circuit breaker
  10. Incident: cascading failure — one slow microservice takes down the platform
  11. Incident: one customer consumes all resources — rate limiting & noisy neighbours

Previous: Incidents II: Scale & Growth · Next: Incidents IV: Correctness, Regions & Operability