Incidents IV: Correctness, Regions & Operability

Duplicate payments, races, deadlocks and locks; consistency choices; AZ and region failures, DR; sudden latency, bad deploys, tracing; slow and stale search

About 115 minutes · 15 exam questions · 9 interview questions · 2 code labs

What you will learn

  1. Incident: duplicate orders and double-charged payments — idempotency
  2. Incident: race condition — oversold inventory and lost updates
  3. Incident: deadlocks in the database
  4. Incident: "we need a distributed lock" — and the lock that failed
  5. Incident: stale reads hurt — strong consistency vs eventual consistency
  6. Incident: one availability zone is down
  7. Incident: a whole region is down — and designing multi-region
  8. Incident: disaster recovery — the accidental DELETE, ransomware, or lost database
  9. Incident: API latency suddenly increases (no deploy)
  10. Incident: errors increase after a deployment
  11. Incident: we cannot find where the latency originates — distributed tracing
  12. Incident: search is slow, or the search index is stale

Previous: Incidents III: Async, Dependencies & Resilience · Next: Real-World Dissections: Famous Architectures & Outages, Explained