Incidents IV: Correctness, Regions & Operability
Duplicate payments, races, deadlocks and locks; consistency choices; AZ and region failures, DR; sudden latency, bad deploys, tracing; slow and stale search
About 115 minutes · 15 exam questions · 9 interview questions · 2 code labs
What you will learn
- Incident: duplicate orders and double-charged payments — idempotency
- Incident: race condition — oversold inventory and lost updates
- Incident: deadlocks in the database
- Incident: "we need a distributed lock" — and the lock that failed
- Incident: stale reads hurt — strong consistency vs eventual consistency
- Incident: one availability zone is down
- Incident: a whole region is down — and designing multi-region
- Incident: disaster recovery — the accidental DELETE, ransomware, or lost database
- Incident: API latency suddenly increases (no deploy)
- Incident: errors increase after a deployment
- Incident: we cannot find where the latency originates — distributed tracing
- Incident: search is slow, or the search index is stale
Previous: Incidents III: Async, Dependencies & Resilience · Next: Real-World Dissections: Famous Architectures & Outages, Explained