Error budgets

Error budgets tell a team whether it is focusing on the right things. A service level objective (SLO) says how reliable a service should be, for example 99.9% of requests answered within a second. The error budget is the room left: the 0.1% that may fail. While the budget lasts, the team can ship features; when it runs out, the team stabilises first.

Setting up

1. Connect a monitoring backend

Under Settings → Error budgets, add the system that holds your metrics:

• Connect Google Cloud Monitoring
• Connect AWS CloudWatch

2. Add a service

On the Error budgets page, open the service menu and choose Add service. Give it a name, a description and the data source (backend) it is measured in. A service can be linked to a team on the Teams page, so the team's reliability is scored over it.

3. Add SLOs

Select the service and press Add SLO. Each SLO has:

• Name, environment and goal, for example 99.5%.
• Step: the window over which the goal is measured, such as 1 hour, 24 hours or 7 days. Agile Analytics computes every SLO every 30 minutes for each step.
• Method: how good and bad events are told apart (see below).

SLOs can be disabled and enabled again without losing their history.

4. Link repositories

In Manage services, pick the repositories that deploy each service. A deployment that is followed within an hour by a fast-burn alert on that service counts as a failed change in the DORA metrics, and the time until the next deployment as its time to restore.

Methods

Good/bad ratio

The share of good events among valid events. Give two of three filters: good, bad and valid. Examples: successful responses over all responses (availability), acknowledged messages over all messages (queue coverage).

Distribution cut

The share of events below (or above) a threshold in a distribution metric, typically latency. Give the valid filter, the threshold bucket and whether a value is good below or above it. The threshold bucket refers to the buckets of the distribution metric in your backend; this sheet lists the bucket boundaries of Google Cloud Monitoring's default latency distribution.

Reading the page

For each SLO you see the goal and the measured value for the period, the error budget left, the burn rate (how fast the budget is being spent; a rate above 1 means it runs out before the end of the window) and a chart of good and bad events over time. The step selector switches between the windows. Notes can be added to a point in time, and a Jira ticket can be attached to an incident.

The last job status at the top tells you whether the last measurement succeeded. When it failed, the data may be stale: check the backend's credentials under Settings → Error budgets.

Where error budgets are used

• Reliability, one of the eight health areas: each SLO scores 100 within its budget, less the further it overspent.
• The Reliability tile on the overview and the team health ring.
• Signals: a message when an SLO goes over its error budget and when it is back within it.
• Change failure rate and time to restore in the DORA metrics, through the linked repositories.

Filter examples (Google Cloud Monitoring)

Latency (distribution cut)

Filter valid:

project="google-project-name"
resource.labels.module_id="module-name"
metric.type="appengine.googleapis.com/http/server/response_latencies"
(metric.labels.response_code = 429 OR metric.labels.response_code = 200 OR
  metric.labels.response_code = 201 OR metric.labels.response_code = 202 OR
  metric.labels.response_code = 203 OR metric.labels.response_code = 204 OR
  metric.labels.response_code = 205 OR metric.labels.response_code = 206 OR
  metric.labels.response_code = 207 OR metric.labels.response_code = 208 OR
  metric.labels.response_code = 226 OR metric.labels.response_code = 304)

Threshold bucket: 19 · Good below threshold: yes

Availability (good/bad ratio)

Filter good:

project="google-project-name"
metric.type="appengine.googleapis.com/http/server/response_count"
resource.type="gae_app"
resource.label.module_id="module-name"
(metric.labels.response_code = 429 OR metric.labels.response_code = 200 OR
  metric.labels.response_code = 201 OR metric.labels.response_code = 202 OR
  metric.labels.response_code = 203 OR metric.labels.response_code = 204 OR
  metric.labels.response_code = 205 OR metric.labels.response_code = 206 OR
  metric.labels.response_code = 207 OR metric.labels.response_code = 208 OR
  metric.labels.response_code = 226 OR metric.labels.response_code = 304)

Filter valid:

project="google-project-name"
metric.type="appengine.googleapis.com/http/server/response_count"
resource.type="gae_app"
resource.label.module_id="module-name"

Queue coverage (good/bad ratio)

Filter good:

project="google-project-name"
metric.type="pubsub.googleapis.com/subscription/ack_message_count"
resource.type="pubsub_subscription"

Filter bad:

project="google-project-name"
metric.type="pubsub.googleapis.com/subscription/num_undelivered_messages"
resource.type="pubsub_subscription"