Reliability targets your team can actually run to
Service level objectives, error budgets, a clear incident process and blameless postmortems, set up so your team is paged rarely and for real reasons.
Overview
Reliability work goes wrong in two ways. Either nobody agrees what good looks like, so every blip is an emergency, or alerts fire so often that people stop reading them. Service level objectives fix the first. Better alerting and design fix the second.
We work during business hours and do not sell on-call. Instead, we help you build systems and processes where night-time pages are uncommon: alerts tied to user impact, automatic recovery for known failures, and postmortems that lead to fixes rather than blame.
Signs you need this
- Engineers are woken up for alerts that need no action
- Product and engineering argue about reliability without shared numbers
- Incidents are handled differently every time, depending on who is around
- The same outage has happened more than once
What you get
- Service level objectives
- Objectives for your user-facing journeys, defined with product owners, with the queries that measure them.
- Error budget policy
- A one-page agreement on what happens when a service spends its budget, signed off by engineering and product leads.
- Alert rework
- Noisy alerts removed or downgraded, and paging alerts rebuilt around burn rate against your objectives.
- Incident process
- Roles, severity levels, communication templates and a channel workflow your team rehearses at least once.
- Postmortem practice
- A blameless template, a review routine and a tracker that makes sure follow-up actions get done.
- Toil reduction backlog
- A ranked list of manual fixes and repeat incidents to automate, with the first few done during the project.
How the work runs
- 1
Reliability review
1 to 2 weeks
Study recent incidents, current alerts and on-call load, and interview the people carrying the pager.
- 2
Objectives and budgets
2 to 3 weeks
Define user journeys, write the measurement queries, agree targets and draft the error budget policy.
- 3
Alerting and incident process
2 to 4 weeks
Replace noisy alerts, set up burn-rate paging, and introduce the incident and postmortem routines.
- 4
Embed and hand over
2 to 4 weeks
Run the new process alongside your team through real incidents and reviews, then step back.
Timings depend on the size of your estate. The assessment fixes them before any build starts.
What you can check afterwards
- Each critical user journey has a written objective and a live dashboard showing its error budget
- Every paging alert links to a runbook and maps to user impact
- Postmortems are written for significant incidents and their actions are tracked to completion
- Your on-call engineers can see a clear drop in pages that needed no action, using your own paging history
Tools we work with
- Prometheus
- Grafana
- Sloth
- Pyrra
- PagerDuty
- Opsgenie
- incident.io
- Kubernetes
- Terraform
- Datadog
Questions buyers ask
- Will you join our on-call rota?
- No. We work business hours only. Our job is to make your rota quieter and your incidents shorter, not to carry the pager for you.
- Do we need a specific monitoring tool?
- No. We work with what you have, whether that is Prometheus, Datadog, CloudWatch or Azure Monitor, and only suggest a change if the current tool cannot measure your objectives.
- How involved does our team need to be?
- Quite involved. Objectives and budgets only stick if your engineers and product owners agree them, so expect a few hours a week from a lead engineer and short sessions with product.
- What happens after the project?
- Your team owns the objectives, alerts and process. We suggest a quarterly review of objectives, which you can run yourselves or ask us to join.
Talk to us about site reliability engineering.
Tell us what you run and where. We reply within one business day with questions or a quote.