Table of Contents

We are around the corner of Black Friday and Holiday season, every SRE’s favorite season, because tensions are at an all-time low (wink wink) and Christmas Spirit is all around us.

Also, there is something else that tends to happen around this time - you guessed it - people are reviewing their runbooks! Hurray!

Runbooks are very peculiar artifacts in our industry, as they often suffer from extreme negligence and most are written without people pouring their hearts into the them, which makes them said and abandoned, never to be read.

And that just breaks my heart, which is usually cheerful at this time of year, because you guessed it, low stress levels (wink wink).

So I wanted to sit down and write an opinionated short guide on how to make “good” runbooks. Runbooks that people will actually read and come back to. Runbooks that they will want to contribute to, never letting them be obsolete. Runbooks that people can follow at 3am, when the poop hits the fan.

Pour Your Heart Into It!

To write a good runbook, you have to care. If you are not interested in the topic or don’t care, then perhaps this runbook shouldn’t exist OR it should exist in a different format. For example, you are writing a generic runbook called XYZ Service because someone told you we must have runbooks before the go-live date.

But what you should be writing are perhaps these, real-life scenarios:

  • XYZ Service - How to restore from distributed lock errors
  • XYZ Service Troubleshooting - Pods restarting with error "Unable to startup..."
  • XYZ Service - How to extract marketing report from users table

So it could be that you just need to think better on the usefulness and audience of the document you are writting.

It’s also funny, how every runbook is written by someone calm, in daylight, who already understands the system. Yet runbooks are mostly read by someone half-asleep, mildly panicking and has been on the team for three weeks.

The test that matters

Before any advice, the single criteria that tells you whether a runbook is good:

Could you walk outside and ask someone from the street to reason about what your runbooks is about and how to use it?

I know that’s a bit extreme but it does help to ground us and to make us KISS. If we keep remind ourselves of that criteria, it will help us remove all the fluff and add all the important bits. The end resullt should be something that we could give to a competent engineer with no prior context and have them be able to reason about it.

Most runbooks fail this badly, and they fail in a specific way: they’re written as reference material when they needed to be explicit instructions. Reference explains how a thing works. Instructions tell you what to do next. At 3am nobody wants to understand the architecture - they want the bleeding to stop.

Runbooks vs Troubleshooting Guides

To make your runbook good, it needs to follow the-mother-of-all-rules for runbooks:

One document, one problem. “Database runbook” is not a runbook, it’s a folder. “MySQL connection pool exhausted” is a starting to look like a runbook. However, I would take this one step further, because even this could be ambiguous. Why?

Because I am not sure if it’s a runbook OR a troubleshooting guide. There is a difference.

  • Troubleshooting Guide - How to figure out what’s wrong or what’s happening with some particular service, or a user problem or an error log
  • Runbook - How to get a specific result by following a series of steps. How to fix some issue (you already know that’s the issue)

KISS

The best runbook I ever used was ugly. Numbered steps, exact commands, max 2 diagrams. It read like IKEA assembly instructions, and that’s precisely why it worked. And it was beautiful (chef’s kiss)

Every extra paragraph is something the reader has to skip past while production is down. A runbook is not documentation of your system - it’s a script for a specific bad situation. It could and should have a reference to your architecture, at most.

More important guidelines:

Commands you can copy. Not restart the affected service but:

kubectl -n payments rollout restart deployment/personalization-api

The reader should never have to translate intent into syntax. Every translation is a chance to get it wrong under pressure, and a chance to hesitate.

Say what you expect to see. This is the most commonly skipped thing and the most valuable:

kubectl -n ecommerce get pods -l app=personalization-api
# Expect: all pods Running, 0 restarts in the last 5 minutes.
# If pods are CrashLoopBackOff, go to step 7 instead.

Without the expectation, the reader runs the command, sees output, and has no idea whether it’s good news. You’ve given them a task instead of a decision.

Write the branches down. Real incidents fork. If X, do this. If Y, go to step 9. Flat linear runbooks quietly assume the happy path, which is exactly the path you’re NOT on.

The meat

If you know the good ol’ hamburger theory in writing, this is the meat of the article, so pay attention :)

A good runbook (or a troubleshooting guide) should have the following:

Title

First, a title that matches the alert. The reader arrived here from a page. The title should be the thing the page said, near enough word for word. Don’t make them wonder if they’re in the right document. This document is linked to you alert, of right? And you are not creating alerts without runbooks to acompany them, right? Right?

Impact

Second, explain the impact, in one line. What is broken for whom, right now. “Customers cannot complete checkout. Revenue-affecting”. This is first because it’s how the reader calibrates everything else, whether to wake someone up, whether to skip diagnosis and go straight to mitigation.

Context/Background/Prerequisites

Third, some context and background, the reader must acknowledge that his circumnstances are the same as explained in the runbook and he can understand better what’s going on. Ideally, you will link your Troubleshooting Guides to your Runbooks and they can be used independently or hand-in-hand, depending on the situation.

If your runbook requires a particular access to some tool, API credentials or something like that, make sure you put that at the top under “prerequisits” so that the person opening the document know if they can execute all the steps or not. Ideally, all your team members have all the accesses but I could tell you some stories…

Sometimes, for very critical issues, you can also put the mitigation at the top, if it’s straightforwrad and known. For example:

## Stop the bleeding

If error rate is above 5% and you are unsure why, roll back first:

    kubectl -n revenue rollout undo deployment/payment-api

Expect error rate to drop within 2 minutes. Verify on <dashboard link>.
Then continue to diagnosis below - but the incident is no longer urgent.

Decision tree and default case

When your system evolves and becomes complex, you will probably need to refactor your runbook and move on from simple structure of numbered steps and make it more like a decision tree, pretty formatted for human eyes. For example:

## Diagnosis

1. Check whether this correlates with a deploy.
   `kubectl -n payments rollout history deployment/checkout-api`
   → Deploy in the last 30 min? Almost certainly the cause. Roll back (above).
   → No recent deploy? Continue to 2.

2. Check dependency health: <link to dependency dashboard>
   → Payment provider latency elevated? Go to "Provider degradation".
   → All dependencies healthy? Continue to 3.

3. Check connection pool saturation: <link to query>
   → Pool at 100%? Go to "Pool exhaustion".
   → Pool healthy? Escalate to team Pikachu - this is not a known failure mode.

That last line is your default case in this switch statement and you should always have it. Because if someone uses the document and doesn’t get help, you need to keep guiding them until they are able to get to someone that can resolve the issue. And that is also fine, because at the time of creation of the runbook, we cannot forsee all possible problems, especially if the system is new.

Escalation with names and numbers

Who to wake, how, and - the part people forget - when it’s acceptable to. Give explicit permission:

If not resolved within 50 minutes, escalate. You do not need to justify escalating.

New team members will suffer for an hour rather than wake someone senior, because nobody told them it was allowed. Tell them.

Verification

How do you know it’s actually fixed? A specific check, not “confirm the service is healthy”:

## Verify

- Error rate below 0.5% for 5 consecutive minutes: <dashboard link>
- One successful API response: `./scripts/verify-api.sh production`

Links to all the relevant context, architecture, tickets, etc… for whoever reads this afterwards. It belongs last because it’s the part that isn’t urgent.

Runbooks rot, and you have to plan for it

An out-of-date runbook is worse than no runbook, because it costs time before it fails. A missing runbook at least tells the truth immediately.

Some things that actually help:

Keep them in git, next to the code. They get reviewed with the change that affects them, and “did you update the runbook?” becomes a PR comment rather than a good intention. This is the docs-as-code idea and it’s the single highest-leverage habit here.

Link from the alert, always. Every alert should carry a runbook_url annotation. There is a dedicated circle in hell for people creating alerts without runbooks, JFYI.

annotations:
  summary: "Checkout error rate above SLO burn threshold"
  runbook_url: "https://github.com/org/repo/blob/main/runbooks/checkout-error-rate.md"

Exercise them. The only reliable way to find a wrong runbook is to follow it. Game days, chaos sessions, or just having whoever is on-call read one at random each week and file a PR for whatever’s wrong. Following it is the test - reviewing it is not, because a reviewer with context will fill in gaps unconsciously.

Let the new person write it. Counterintuitive and very effective. Whoever joined most recently is the only one who can still see what’s missing, because they haven’t yet absorbed the assumptions. Have them write or rewrite it after shadowing an incident; the expert’s job is to correct it, not to author it.

Date the last verification, not the last edit. “Last followed during an incident: 2026-08-14” tells you far more than a git timestamp. A file edited last week might still be describing a system that changed a year ago.