Ridgeline Outfitters · 41 minutes, eight stops

The 9 March outage, minute by minute

scroll = fly forward · a minute is a metre · J next stop · K back

Numbers — six pillars, each standing beside the stop that earned it

Time to detect
— min
Time to first wrong guess
— min
Time to root cause
— min
Downtime
0 min
Orders lost (estimated)
0 orders
Follow-up actions
0 actions

Items — eight stops along the path

  1. STOP 01The checkout certificate expires. Nobody is told.who: nobody · 0 min in
  2. STOP 02Checkout errors spike; the payments log fills with handshake failures.who: the log · 6 min in
  3. STOP 03Pager fires. Priya opens the dashboard on a phone.who: Priya · 9 min in
  4. STOP 04First guess: the CDN. Ten minutes lost on the wrong panel.who: Priya · 13 min in
  5. STOP 05"It's the cert." Expiry date read out loud.who: Marcus · 22 min in
  6. STOP 06Renewed, deployed; the first successful checkout at 09:33.who: Marcus · 29 min in
  7. STOP 07Error rate back to baseline. Incident closed.who: Priya · 41 min in
  8. STOP 08Postmortem: expiry alerts at 30, 14 and 3 days; auto-renew on.who: everyone · 88 min in, past the fog

Story The wall

Nobody was told because nothing was watching. The certificate had a date on it, and the date passed. Everything downstream — the spike, the pager, the wrong guess — followed from a silence that a single alert would have broken a month earlier.

The wrong guess cost ten minutes and is the beat worth remembering. When the pager fires, the first move is now "what expired?", asked out loud, before anyone opens a panel.

Written on the wall · Priya · 09:02, before

Steps — what changed, laid as floor tiles

  1. Steps · what changed · 09:02, nothing yet
  2. Certificate expiry alerts at 30, 14 and 3 days, to #ops and the pager.
  3. Auto-renew on for every certificate we own.
  4. The incident runbook opens with "what expired?"
  5. Pager fires at 3 min of checkout errors, not 9.
  6. The 38 customers who hit the error got an email and a code.

Table The slab · time, what, who

TimeWhatWho
09:02cert expiresnobody
09:11pagerPriya
09:24root causeMarcus
09:43closedPriya

Code — the terminal pane

basecamp · 09:02tty · noir
basecamp certs list --expiring 30d
basecamp alerts add cert-expiry --days 30,14,3
basecamp incident timeline 2026-03-09     # 8 stops · 41 min
$
“Nobody was told because nothing was watching.”
Ridgeline Outfitters is fictional. Every number on this page is placeholder data.
Camera in 3D · INVESTIGATION · colorway Noir · all 29 worlds
09:020 min in · stop 1 of 14
scroll = fly · mouse = look · shift+wheel = dolly
    exit