Production Observability: Signals, Service Objectives, and Useful Alerts

Observe the service from the user's perspective

Operational visibility is useful when it helps answer whether users can complete their work and what is preventing them when they cannot. A host CPU graph is not a service health definition. An API might be healthy from a CPU perspective while its database pool is exhausted and requests time out.

Start with a small set of service signals: request rate, errors, latency distribution, and saturation of constrained resources. Break these down by meaningful operation or dependency, but avoid unbounded labels such as user IDs that create high-cardinality metrics and costly storage.

Percentiles and objectives

An average latency can look fine while a meaningful fraction of requests are painfully slow. Track percentiles such as p50, p95, and p99 with enough volume and an appropriate histogram design. Pair these measurements with a service-level objective (SLO), such as the proportion of eligible requests completed successfully within a defined latency threshold over a rolling window.

An error budget is the remaining allowance for missing that objective. It gives product and engineering a shared way to discuss reliability work against feature delivery. An SLO is most useful when its definition matches user impact and teams can take action when the budget burns quickly.

Alerts should page for action

An alert should indicate a user-impacting problem or an imminent risk that requires a human response. Alerting on every threshold crossing creates fatigue. Prefer multi-window burn-rate alerts for SLO violations, and route lower-urgency capacity trends to dashboards or tickets.

Every page should answer: what is affected, how severe is it, what changed, and where is the runbook? If no reasonable action follows the alert, it is probably a chart annotation rather than a page.

Use telemetry together

Metrics show aggregate patterns and timing. Logs provide discrete contextual events. Traces connect spans across a request path and reveal where time was spent. Carry a request or trace identifier across boundaries, but take care not to log credentials, personal data, or entire payloads by default.

A practical rollout

Instrument one critical user journey end to end. Define its success and latency objective, validate dashboards with a controlled failure, and rehearse the alert response. Expand only after the signals help an on-call engineer decide what to do. Observability is a feedback system for operating software, not a contest to collect the most telemetry.