Production Judgment: Avoiding Fallback, Constant Work, Queue Backlogs, Metastable Failures & Rollback-Safe Deploys

The senior-level instincts from the Amazon Builders' Library and metastable-failure research: why rarely-run code paths betray you, how constant-work designs remove modes, how queues fall into insurmountable backlogs, why outages outlive their triggers, and how to make every deploy reversible.

About 110 minutes · 17 exam questions · 10 interview questions · 2 code labs

What you will learn

  1. The production-judgment lens: modes are where systems die
  2. Avoiding fallback: why the backup plan is usually the outage
  3. What to do instead of fallback: reliable primaries, pushed data, always-on failover
  4. Constant work: the coffee urn, Route 53 health checks and Hyperplane
  5. Applying constant work in Node: level-triggered reconcilers that heal themselves
  6. Queue backlogs are bimodal: the recovery math nobody does up front
  7. Backlog-tolerant queue design: fresh-first, drop the superseded, isolate the noisy
  8. Consumers under backlog: leases, heartbeats, in-flight limits and Little's law
  9. Metastable failures: when the outage outlives its trigger
  10. Breaking metastable loops: characteristic metrics, hidden capacity and goodput-first policies
  11. Rollback safety: two-phase prepare/activate for formats and protocols
  12. Hands-off deployments: one-box, waves, bake time and alarm-driven rollback

Previous: Rate Limiting & Throttling