Keeping a Runbook True

Runbooks · July 9, 2026 · 9 min read

The moment that exposes a runbook is always the same. Somebody opens it during a real event, follows it to step four, and finds an instruction to click something in a console that was decommissioned two quarters ago. The document is not wrong because anybody was careless. It is wrong because it was a copy of the truth, and the truth moved.

That property determines what can be done about it. Runbook decay is not a discipline problem to be solved by reminding people to update documentation. It is structural: a runbook describes a system it is not connected to, and nothing enforces agreement between the two. Code referencing a function that no longer exists fails to build. Prose referencing a console that no longer exists reads perfectly well.

Three kinds of content, three rates of decay

Most runbooks mix three sorts of material with wildly different lifespans, and mixing them is what makes maintenance feel hopeless.

Intent — why this procedure exists, what condition it addresses, what outcome counts as success, when not to proceed. This decays slowly. It survives migrations, re-platformings and reorganisations, because it is a statement about the problem rather than the machinery.

Structure — which components are involved, what depends on what, which constraints must hold while the work is done, who has the authority to approve the irreversible parts. This decays at the pace of architectural change: noticeably, but on the scale of quarters.

Mechanism — the exact commands, hostnames, menu paths, field names, ticket queues and screenshots. This decays continuously and invisibly, and it is typically eighty per cent of the document.

Separating the three makes maintenance tractable. Intent and structure are written once and revisited rarely. Mechanism should be generated, linked or executed rather than transcribed — pulled from the system that owns it, so that it is either current or obviously absent. The usual arrangement is the worst one: a paragraph of durable reasoning buried inside four pages of instructions that expired without notice, so the whole document is distrusted equally.

The only reliable detector is execution

You cannot review a runbook into correctness. Reading it confirms that it is coherent, not that it corresponds to anything; reviewers reconstruct the missing steps from their own knowledge without noticing, which is precisely why the author could not see the gaps either.

Execution is the test. Somebody who does not already know the answer follows the document, exactly as written, and records every point where reality and text diverge. The deviations are the output — not a corrected document but a count, and that count is the only health metric a runbook has.

Three ways to obtain executions, in ascending order of cost and value:

• Rehearsal. Scheduled, deliberate, in a safe environment, with a person who is unfamiliar with the procedure holding the keyboard and the person who wrote it forbidden from speaking.
• Opportunistic real use. When a genuine low-severity event happens to match a documented procedure, use the document instead of the knowledge in somebody’s head, and capture the divergences afterwards.
• Exercise under pressure. The full rehearsal with the clock running, the communications obligations included, and the people who would really be involved actually involved.

The second of these is undervalued. It costs almost nothing and exercises the document against real conditions, and its main obstacle is cultural: the fastest route through a minor event is to let the expert do it from memory, which is exactly the choice that leaves the document rotting for the major one.

Bind the document to the thing it describes

Decay accelerates when the people changing a system have no way of knowing that a procedure depends on it. They are not neglecting the runbook; they have never seen it and would have no reason to look. The fix is to create a link that somebody making a change will actually encounter.

Reference the runbook from the configuration or code that implements the thing, so it appears in the diff of anyone touching it. Give each runbook a stable identifier unique enough to be searched for, and use that identifier in alert definitions, change records and incident notes, so everything affected by a change can be found with a text search. Record in the runbook itself which systems it assumes — as a list, not as prose — so the reverse lookup is possible at all.

None of this makes the runbook update itself. It converts a silent dependency into a visible one, which is the difference between decay you discover during an incident and decay you discover during a change.

Deleting is maintenance

Every operations team under audit pressure accumulates procedures, because a document that exists is evidence and a document that does not is a finding. The result is a library in which most items have never been executed, none has been retired, and nobody can tell the three that are current from the forty that are archaeology.

An untested runbook is worse than nothing, because nothing prompts a person to think whereas a confident obsolete document prompts them to act. Under time pressure a plausible document carries an authority it has not earned, and that is how procedures cause incidents rather than resolving them.

Two habits keep the library honest. Give every runbook a review date and move the expired ones somewhere visibly outside the operational set — archived, not deleted, but unmistakably not live. And be willing to say in writing that a scenario has no procedure and will be handled by improvisation with a named escalation. A candid gap is safer than a fictional control, and it is the version an experienced assessor believes.

Write for the worst moment it will be used

The reader you are writing for is not you, is not rested, has been awake for twenty minutes, and is reading on a phone before they reach a keyboard. That reader needs a different document from the one that gets written when someone tidies up their notes after an event.

Put the decision first. Before any steps, state the condition this procedure applies to and — more importantly — how to recognise that it does not apply. The most valuable line in a runbook is the exit ramp: if what you are seeing does not match this description, stop here and escalate. Its absence is what turns a misdiagnosis into a procedure confidently applied to the wrong problem.

Then the preconditions, including the authority required for anything that cannot be undone. Then the steps, each with its expected result, so a divergence is detected at the step that caused it rather than three steps later. And at the end, how to tell that it worked, and what to do if the outcome is ambiguous — the state real procedures produce far more often than clean success or failure.

Do not automate what you have not run

The final temptation is to convert the procedure into a script and consider the maintenance problem solved. It is not solved; it is hidden. An automated procedure that has never been executed by a human is a set of assumptions compiled into a form that is harder to inspect and much harder to abandon halfway through.

Automation is the right destination for procedures run often enough that the variations are known and the decision points are genuinely mechanical. Applied earlier, it freezes today’s understanding into a system that will keep confidently performing today’s steps long after they have stopped being the right ones — and will perform them faster than anybody can intervene.