Observability and SLO engineering

A useful service-level indicator is close to the boundary where the user observes the result. Measure accepted transactions that reach a durable confirmed state, not the health of one internal method. An implementation can move from a monolith to services without changing the meaning of successful submission. Internal metrics still matter for diagnosis, but they should not redefine reliability whenever the component graph changes.

Common indicators include availability, latency, correctness, freshness and completeness. An asynchronous workflow may need two clocks: whether intake was accepted promptly and whether processing completed within its target. A data feed may be reachable while serving stale records, so freshness belongs beside request success. Pick the few indicators that reveal user harm. A dozen weak objectives create reporting work without guiding a decision.

Average latency hides slow users. Use percentiles or threshold compliance. A statement such as the share of valid requests completed within a defined duration is easy to connect to an objective. Keep the measurement point consistent. Browser timing includes network and rendering. Gateway timing includes more of the dependency path than application timing. Each can be valid, but they answer different questions.

Watch low-traffic services. A percentile calculated over a handful of requests behaves strangely, and one failure can dominate a short window. Longer windows, synthetic checks or event-based objectives may work better. Also separate fast failures from success. An endpoint that rejects every request in a few milliseconds has excellent latency and no utility. Latency indicators should normally apply to successful outcomes, with availability or correctness capturing the rejected work.

If an objective permits a small fraction of valid events to fail during a window, that allowance is the error budget. The budget gives product and operations teams a shared way to discuss release risk. Healthy budget allows ordinary change. Fast budget burn calls for investigation, reduced rollout pace or reliability work. The exact policy is an organisational choice, but it should be decided before a disputed release.

Error budgets are not permission to ignore individual incidents. A privacy breach or duplicated settlement may require action even if its count fits inside an availability budget. Use separate controls for low-frequency, high-consequence events. Also avoid turning the objective into a public contract without examining traffic, dependency and support assumptions. An internal reliability target, a support agreement and a legal SLA can use related indicators while carrying different remedies and exclusions.

An alert based on one failed request is noisy. An alert that waits until the whole monthly budget is gone is late. Burn rate compares current error consumption with the rate that would spend the budget evenly across its window. Fast burn over a short window catches severe incidents. Slower burn over a longer window finds persistent degradation. Pairing windows reduces sensitivity to momentary spikes.

Route alerts according to the action they require. A page should mean somebody can reduce user harm now. A ticket can capture a slower trend. A dashboard is not an alert queue. Every page needs a runbook link, the affected journey, current budget state and enough dimensions to locate the failure. Alert text that reports only a pod name sends responders into infrastructure before they know whether users are affected.

Metrics aggregate rates and distributions efficiently. Logs preserve discrete events and business context. Traces connect work across service boundaries. None replaces the other. A reliability signal may come from a metric, while a trace identifies the slow dependency and a structured log records why a policy rejected the transaction. Design the correlation identifiers and field names before an incident so responders can move between them.

Cardinality needs control. Putting account, user or transaction identifiers into metric labels can overwhelm the time-series store and expose sensitive data. Keep high-cardinality identifiers in restricted logs or traces. Metrics use bounded dimensions such as route, result class, dependency and region. Sample traces according to value: retain errors and unusual latency more aggressively, but make sampling decisions visible so an absent trace is not mistaken for an absent request.

Asynchronous systems need more than queue depth. Report accepted rate, completion rate, oldest unprocessed event, processing delay, retry count and dead-letter age. A queue of ten recent events may be fine. One event stuck for hours can block a specific account or partition. Measure age by meaningful key where possible, and expose consumer position so replay or stalled ownership can be diagnosed.

Scheduled work needs expected-start and expected-completion signals. A job that never launches produces no error log inside its own code. An external scheduler or heartbeat must notice the missing execution. For data pipelines, record input watermark and output watermark. Freshness then reflects how far the result trails the source event time, not merely when the last process wrote a row.

A user does not care which provider caused the failure. The service team still needs to separate dependency health for response and contract management. Track calls, timeouts, rejection and latency by dependency. Use bounded retries and circuit breakers so a slow provider does not consume every worker. When the user journey can degrade safely, measure the degraded result as its own outcome rather than calling it fully healthy.

Third-party exclusions should be narrow. If the product cannot operate without a payment, identity or messaging service, removing all provider failures from the objective hides the experience. A layered model works better: measure the end-to-end journey, then use component and dependency indicators to assign cause. Commercial SLAs can identify responsibility, but operational telemetry should still show the complete path.

During an incident, responders need a timeline of user impact, detection, decisions and recovery. Stable deployment identifiers, feature flags and configuration versions help connect a change to the burn. Record mitigation actions and their observed effect. A rollback that coincides with recovery is evidence, though not always proof of cause. Keep the timeline separate from speculation until the team can test it.

The review should improve a control, not merely blame a missing alert. Ask why the system allowed the failure to spread, why detection took as long as it did and why recovery required the steps it did. Actions need owners and verification. A new dashboard nobody checks is not a fix. A bounded retry, safer release gate, tested restoration path or changed objective can be verified against the scenario that exposed the gap.

Store indicator queries, objective thresholds, alert rules and ownership with version history. Test queries against known fixtures, because a syntax-valid query can measure the wrong denominator. Review objectives when traffic shape, dependencies or user journeys change. A target inherited from an old architecture may reward the wrong behaviour after a product redesign.

Start with one important journey. Instrument it at the user boundary, define the objective, create burn alerts, write the runbook and run a failure exercise. That small loop exposes missing identifiers and unclear ownership faster than a company-wide telemetry rollout. Observability becomes useful when it changes a decision: page now, stop a rollout, open capacity, contact a provider or accept that the service is within its agreed operating range.