Incidents III: Async, Dependencies & Resilience
Growing queues, slow and crashing consumers, duplicates, ordering, retries and retry storms, unavailable or slow downstreams, circuit breakers, cascading failures and noisy tenants
About 115 minutes · 15 exam questions · 9 interview questions · 2 code labs
What you will learn
- Incident: the queue keeps growing — consumer slower than producer
- Incident: consumer crashes during processing (and poison messages)
- Incident: duplicate messages — "the customer got charged twice"
- Incident: message ordering required — "refund processed before the payment"
- Incident: we need a retry mechanism (and the naive one made it worse)
- Incident: retry storm — the downstream recovered but we are still down
- Incident: a downstream is unavailable
- Incident: a downstream is slow (worse than down)
- Incident: we need a circuit breaker
- Incident: cascading failure — one slow microservice takes down the platform
- Incident: one customer consumes all resources — rate limiting & noisy neighbours
Previous: Incidents II: Scale & Growth · Next: Incidents IV: Correctness, Regions & Operability