Ridgeline Outfitters · 41 minutes, eight beats

The 9 March outage, minute by minute

Numbers

Downtime
0 → 41 min
Orders lost (estimated)
0 → 38
Time to detect
— → 6 min
Time to first wrong guess
— → 13 min
Time to root cause
— → 22 min
Follow-up actions
0 → 5

Items Eight trees, eight beats

  1. The checkout certificate expires. Nobody is told.who: nobodyBud
  2. Checkout errors spike; the payments log fills with handshake failures.who: the log · 6 min to detectBud
  3. Pager fires. Priya opens the dashboard on a phone.who: PriyaBud
  4. First guess: the CDN. Ten minutes lost on the wrong panel.who: Priya · 13 min to the wrong guessBud
  5. "It's the cert." Expiry date read out loud.who: Marcus · 22 min to root causeBud
  6. Renewed, deployed; the first successful checkout at 09:33.who: MarcusBud
  7. Error rate back to baseline. Incident closed.who: PriyaBud
  8. Postmortem: expiry alerts at 30, 14 and 3 days; auto-renew on.who: everyone · at the shrineBud

How to read the print

A tree in bud is a beat that has not happened yet.

A tree in full bloom is the beat happening now — the boat on the river is under it.

A tree shedding petals is a beat that is over. The petals land on the water: every petal on the river is an order lost.

The river is the 41 minutes, 09:02 at the left bridge to 09:43 at the right one. The stones under trees two, four and five carry the minutes to detect, to the wrong guess, and to root cause. Each lantern is who was there; it lights when their beat comes. The five wishes tied at the shrine are the follow-up actions.

Story The scroll

Nobody was told because nothing was watching. The certificate had a date on it, and the date passed. Everything downstream — the spike, the pager, the wrong guess — followed from a silence that a single alert would have broken a month earlier.

The wrong guess cost ten minutes and is the beat worth remembering. When the pager fires, the first move is now "what expired?", asked out loud, before anyone opens a panel.

Steps What changed

  1. Certificate expiry alerts at 30, 14 and 3 days, to #ops and the pager.
  2. Auto-renew on for every certificate we own.
  3. The incident runbook opens with "what expired?"
  4. Pager fires at 3 min of checkout errors, not 9.
  5. The 38 customers who hit the error got an email and a code.
Brushed · Priya · 09:02, before
印

Table Ema board · time, what, who

絵馬 えま · Ema board at the shrineHung 09:02
TimeWhatWho
09:02cert expiresnobody
09:11pagerPriya
09:24root causeMarcus
09:43closedPriya

Code Goshuin · the stamped card

証
御朱印 · basecamp · 09:02
basecamp certs list --expiring 30d
basecamp alerts add cert-expiry --days 30,14,3
basecamp incident timeline 2026-03-09     # 8 beats · 41 min
shrine>
令和八年三月九日の朝
“Nobody was told because nothing was watching.”
exit