Methods
Every engagement uses some mix of four methods. Each one answers a single question about your system.
Who decides
An escalation ladder names who can halt the system, what triggers each step up, and how fast each tier has to meet. Every decision gets an owner and a path upward.
| Tier | Who decides | Trigger | Meets |
|---|---|---|---|
| T1: Operations | On-call engineering lead | Error budget over 50% used | Daily during the incident |
| T2: Governance | Service owner and policy lead | More than 3 P1s in a week | Within 24 hours |
| T3: Leadership | Executive sponsor | Regulator inquiry or data loss | Within 48 hours |
You get the ladder, a short charter leadership can approve, and a checklist of the evidence a board or regulator will ask for.
What warns you first
Early-warning thresholds turn the signals that come before a failure into alerts with an owner: queue depth, collapsing model confidence, a shift in what people complain about.
| Signal | Source | Yellow | Red | Owner |
|---|---|---|---|---|
| API latency p99 | APM dashboard | Over 500ms for 5 min | Over 2s for 1 min | Engineering lead |
| Model output dispersion | Inference logs | 2× baseline for 1 hour | 5× or over 1% empty | ML engineer |
| Complaint sentiment | Support queue | Over 30% negative | Over 60% or legal keywords | Comms lead |
| Staff overtime | Time tracking | Over 12 hours in a week | Over 16 hours | Service owner |
We tune the thresholds against your incident history so they fire on real trouble, not noise.
Has anyone practiced
A failure drill is a timed, scripted rehearsal with the people who would actually be on the call. Example scenario: the primary on-call doesn’t acknowledge a P1 within 8 minutes. The secondary takes command, the comms lead logs the handoff, and the review records why the page was missed.
Other scenarios we run: a week of model drift with pressure to keep shipping, a third-party model degrading without warning, and a regulation changing mid-quarter. We measure minutes to acknowledgement, whether owner changes were written down, and the overtime and workarounds afterwards. The maintenance readiness snapshot shows where to start.
Can it be undone
A rollback kit is agreed before launch, so nobody drafts it during an incident. It holds a rubric for when to slow, stop, or reverse, the messages to send to customers and regulators, and the steps for handling data with an audit trail.
| Signal | Threshold | Action | Owner | Clock |
|---|---|---|---|---|
| Error rate spike | Over 5% for 2 min | Slow (canary only) | Engineering lead | 5 min |
| P1 count | More than 3 in an hour | Stop the deploy | Incident commander | 2 min |
| User-facing data loss | Any confirmed loss | Roll back | Service owner | Immediate |
| Regulator inquiry | Any | Roll back | Policy lead | Immediate |
The stoppability checklist scores how close you are today. For prices and formats, see engagements.