Table of Contents

Every few months someone in a leadership meeting discovers SLOs (Service Level Objectives), and what follows is something along the following lines: A spreadsheet appears where every service gets a row. Every row gets 99.9% if “it’s important” and 99.5 if “it’s not critical” because that looks good, “we will adjust it later”. A dashboard is built, a Confluence page is written, and then nothing happens for about eighteen months until somebody asks why we are still getting paged at 3am for things nobody cares about.

Welcome to SLO theatre. It has all the artifacts of the practice and none of the mechanism.

An SLO is a decision-making tool, not a reporting-to-leadership tool

People treat an SLO as a measurement - something you compute, put on a wall, and feel good or bad about.

What it really is a pre-negotiated agreement about what you will do when things get bad. The number is the trigger. If nothing in your organisation changes when you breach it, you don’t have an SLO. You have a magic number that makes you happy or sad.

So before picking any number, let’s answer this: what will we actually do differently when this is breached?, If the answer is “we’ll mention it in the retro” or something similar, just stop. You are about to do a play in the SLO theatre. Go and get agreement on the consequence first, then pick the number. The consequence is the hard part and everybody skips it because it requires interaction with other people.

Start from the user/client, not from your service

The most common mistake after the spreadsheet is measuring the wrong thing. CPU is not an SLI. Pod restarts are not an SLI. These are causes, and you already alert on them, and that is fine - but they are not what your user experiences.

Ask instead: what does a user do with this system, and how would they know it was broken?

For a checkout API, that’s roughly: did my payment go through, and did it happen fast enough that I didn’t give up? That gives you two candidate SLIs (Service Level Indicators) immediately:

  • Availability - proportion of checkout requests that returned a non-5xx response
  • Latency - proportion of checkout requests served faster than some threshold

SLIs

The format that works is almost always:

good events / valid events

A ratio of good things to total things, both of which you can point at in your data. Avoid the averages because averages hide exactly what you care about - a service can average 200ms while 5% of your users wait nine seconds, and the average will look wonderful the entire time.

Picking a target: work backwards from what you already have

Do not pick a target aspirationally: measure what you currently deliver, then decide whether that is acceptable.

Pull the last four weeks of data and compute what your SLI actually was. You will get one of three outcomes:

99.95%, and nobody is complaining. Congratulations, you have found your target, and it’s probably a bit lower than what you’re delivering. Set it at 99.9% and take the difference as room to move faster.

99.2%, and nobody is complaining. Genuinely interesting. Your users tolerate more than you thought. Set the target near where you are. Do not “improve” it to 99.9% because that looks better on a slide - you will have committed to expensive work that no user asked for.

99.5%, and people are furious. Now you have a real gap, and an SLO that’s worth having. Set the target where the complaining stops and work on maintaining that.

Notice that in two of those three cases, the correct move is to set a target at or below current performance. That feels wrong to most engineers. It isn’t. An SLO is not a stretch goal, it’s the line below which you stop shipping features (or at least it should be). Setting it above what you can deliver just means you will live permanently in breach, everyone stops believing the number, and you’re back to theatre.

Nines, nines everywhere

Every nine you add multiplies cost, and the difference between them is much bigger than it looks:

Target Downtime per 30 days Description
99% 7h 12m A bad afternoon
99.5% 3h 36m A long incident
99.9% 43m One page, handled calmly
99.95% 21m Automated recovery only
99.99% 4m 19s Nobody is reading a runbook in time

That last row is the one to sit with. At four nines, a human being cannot open a laptop and read a runbook inside the budget. Everything must be automatic. That is not a monitoring problem, it is an architecture and headcount problem, and it should be a deliberate, well-funded decision - not the number you added because the service “is critical”.

The error budget

The SLO gives you a target. The inverse gives something more useful.

If target is 99.9% over 30 days, error budget is the other 0.1% - about 43 minutes of failure you are allowed to spend. Spend it on deploys, risky migrations, even a chaos experiment. It is yours, and an unspent error budget is not a prize, we should be targeting to always be very close to the SLO target, not always above.

This orients the conversation and ideally gets you the agreement what reliable is for that particular service, while everyone are calm. Without a budget, that conversation (or argument) happens during an incident, usually between tired people, and gets settled by whoever is most senior (or most upset).

The policy needs to be written down and it can be short:

Budget remaining: We ship normally. Deploy on Fridays if you like.

Budget below 25%: No risky changes. Migrations and infra work need a sign-off.

Budget exhausted: Feature work stops. Reliability work only, until the window rolls forward.

But everyone have to honor this agreement. If a VP can override it by asking, then don’t have an error budget or an SLO. You have some graph with extra steps. And that’s also fine, as long as that is clear from day one and NOT on the day when you need to stop working on new features to fix reliability.

Burn rate & Alerting

No matter how I try to word it or add diagrams, this part will never be as cool as what my ex-colleague has written so I will gladly just let you read that, for much better understanding on how to setup SLO alerting.

Check it out -> https://ervinbarta.com/2021/10/19/slo-alerting-for-mortals/

Implementation

Do not roll this out across forty services. Pick one, ideally one that matters, that you own end to end, and where you’d actually notice if it broke.

Give it one or two SLIs, a target derived from real data, a written budget policy, and burn-rate alerts. Live with it for a quarter.

You will get things wrong. Your first target will probably be too strict, your latency threshold will be arbitrary, and you’ll discover your metrics don’t distinguish “user error” from “our fault”. Fixing those on one service is a week’s work. Fixing them on forty is painful so take your time.

A rough sequence that works:

  1. Write the user journey down in a sentence. No metrics yet. “A customer submits a payment and gets a confirmation quickly.”
  2. Turn it into one or two ratio SLIs. Check you can actually compute them from data you already have.
  3. Measure the last 28 days. Find out what you really deliver.
  4. Set the target. At or slightly below what you measured, unless users are already unhappy.
  5. Write the budget policy. Get someone with authority to agree to it, in writing.
  6. Add two burn-rate alerts. Delete the alerts they make redundant.
  7. Review monthly. Adjust the target, not the reality.

That last point matters. Reviewing means asking was this target right? - not quietly redefining what counts as an error until the graph goes green.

What NOT to do

A short list of the ways I’ve seen this fail, all of which I’d rather you avoided:

  • Too many SLOs. Three per service, maximum. If everything is measured, nothing is prioritised.
  • SLOs on things you don’t control. Don’t put a third-party payment provider’s downtime in your budget unless you can actually do something about it. Either exclude it, or make it a separate SLO with a separate conversation.
  • Measuring at the wrong point. Metrics from inside your own service will happily report 100% while your load balancer returns 503s to everyone. Measure as close to the user as you can get.
  • Silently adjusting the target. If the number is embarrassing, the options are to fix the service or to renegotiate the target openly. Redefining the SLI until the number improves is lying to yourself and everyone else.
  • Treating it as an SRE-team artifact. If the team that operates the service can’t recite the SLO from memory, it isn’t theirs. It’s a report someone else runs.

Outro

In summary and for anyone who scrolled to the end out of boredom:

  • An SLO is an agreed decision rule, not a report. If nothing changes on breach, it’s theatre.
  • Measure the user’s experience as good / valid ratios, not your infrastructure’s vital signs.
  • Derive the target from what you already deliver, not from what sounds impressive.
  • The error budget is the actual product - it turns a political argument into arithmetic settled in advance.
  • Alert on burn rate, then delete the alerts it makes redundant.
  • One service, one quarter. Then expand.

The single question that tells you whether any of this is real: when the budget is gone, does feature work actually stop?

Everything else is a graph.

Have fun :)

Resources