Observability: Logs, Metrics, Traces, SLOs and Alerts

Users are complaining on social media that Pixly is slow. The team's dashboards are green: average CPU is 40%, average latency is 120 ms, error rate is 0.2%. An engineer spends two hours searching logs on individual servers before discovering that one of eighteen API servers has a misbehaving network card, and every request it handles takes two seconds. One in eighteen users is having a terrible time; the averages hid it completely.

A system you cannot see into is a system you cannot run. Observability is the ability to understand what is happening inside a system from what it emits, well enough to answer questions you did not anticipate when you built it. This article covers what to emit, how to connect it, what to alert on, and how to keep the cost under control.

• Monitoring versus observability.
• The three signals: logs, metrics and traces, and what each is good for.
• What to measure: the golden signals, RED and USE.
• Percentiles and histograms, and a common aggregation mistake.
• Distributed tracing and context propagation, and what OpenTelemetry is.
• SLIs, SLOs, error budgets and burn-rate alerting.
• Health checks and synthetic monitoring.
• Observing queues and asynchronous work, and controlling observability cost.

Monitoring answers questions you decided in advance: is CPU above 80%? Is the error rate above 1%? Observability is the broader property of being able to ask new questions when something unexpected happens: why are requests from Android version 7.2 in Mumbai slow since 14:00? OpenTelemetry's observability primer describes it as understanding a system from the outside by examining its outputs, which requires the system to be properly instrumented (Observability primer).

In practice, you get observability by emitting rich, connected signals and having tools to slice them.

OpenTelemetry lists traces, metrics and logs (and baggage) among its stable signals (Signals).

They are most powerful connected: an alert on a metric leads to example traces of slow requests, and each trace links to the logs written while handling it. The glue is shared identifiers, especially the trace id.

{"ts": "2026-10-04T21:02:11.482Z", "level": "ERROR", "service": "feed-api", "trace_id": "4bf92f3577b34da6a3ce929d0e0e4736", "user_id_hash": "a91f...", "route": "GET /v1/feed", "status": 504, "duration_ms": 2003, "dependency": "recommendations", "message": "dependency timeout"}

Structured logs (key-value, usually JSON) can be searched, filtered and aggregated by field. Free-text lines like ERROR something went wrong for user 42 cannot, at scale.

• Do log: errors with context, important state transitions (order paid, payment failed), security-relevant events (login failures, permission changes), and the identifiers needed to connect events (request id, trace id).
• Do not log: passwords, tokens, full card numbers, secrets, or unnecessary personal data. OWASP's logging guidance lists such data among things to exclude, mask or protect (Logging Cheat Sheet).
• Levels (ERROR, WARN, INFO, DEBUG) let you control volume; keep DEBUG off in production by default.

Ship logs to a central system so you can search across all servers. Keep audit logs (who did what, for compliance) separate, with stricter retention and integrity controls (article 28).

• Counter: only goes up (requests served, errors). You look at its rate.
• Gauge: goes up and down (queue length, memory in use, connections open).
• Histogram: counts observations in buckets (how many requests took 0 to 10 ms, 10 to 25 ms, and so on), from which percentiles can be computed and aggregated correctly across servers.

Google's SRE book proposes four golden signals for every user-facing system (Monitoring Distributed Systems):

1. Latency: time to serve requests, with successful and failed requests measured separately (a fast error is not a fast success).
2. Traffic: demand, such as requests per second.
3. Errors: the rate of failed requests, including explicit errors, wrong content, and requests that violate policy (for example, slower than the SLO).
4. Saturation: how full the system is, with emphasis on the most constrained resource.

Two related checklists are widely used:

• RED for services: Rate, Errors, Duration.
• USE for resources (CPU, disks, connection pools, queues): Utilisation, Saturation, Errors.

Add business metrics: orders per minute, uploads per minute, sign-ups. They often reveal problems that technical metrics miss ("errors are flat, but orders have dropped 40%").

Metrics carry labels (route, status, region). Every distinct combination of label values is a separate time series that must be stored. A label with a few values is fine; a label like user_id with millions of values creates millions of series and can overwhelm a metrics system. Put high-cardinality detail in logs and traces, not metric labels.

From article 02: report latency as percentiles. But be careful how you combine them. Suppose one busy, healthy server and one quieter server with a problem each compute their own p99. This Java program shows what happens if you average those per-server percentiles instead of computing the percentile over all requests (which is what histograms let you do).

// File: PercentileDemo.javaimport java.util.ArrayList;import java.util.Collections;import java.util.List;import java.util.Locale;public class PercentileDemo {    static long percentile(List<Long> values, double p) {        List<Long> sorted = new ArrayList<>(values);        Collections.sort(sorted);        int rank = (int) Math.ceil(p / 100.0 * sorted.size());   // nearest-rank method        return sorted.get(Math.max(rank, 1) - 1);    }    public static void main(String[] args) {        List<Long> serverA = new ArrayList<>();   // busy, healthy server: 900 requests        for (int i = 0; i < 900; i++) {            serverA.add(20L + (i % 10));          // 20 to 29 ms        }        List<Long> serverB = new ArrayList<>();   // quieter server with a problem: 100 requests        for (int i = 0; i < 100; i++) {            serverB.add(i < 50 ? 30L : 2_000L);   // half of them take 2 seconds        }        List<Long> all = new ArrayList<>(serverA);        all.addAll(serverB);        long p99A = percentile(serverA, 99);        long p99B = percentile(serverB, 99);        System.out.println("p99, server A:            " + p99A + " ms");        System.out.println("p99, server B:            " + p99B + " ms");        System.out.println("average of the two p99s:  " + (p99A + p99B) / 2 + " ms (meaningless)");        System.out.println("true p99, all requests:   " + percentile(all, 99) + " ms");        System.out.println("true p95, all requests:   " + percentile(all, 95) + " ms");        double mean = all.stream().mapToLong(Long::longValue).average().orElse(0);        System.out.printf(Locale.US, "mean, all requests:       %.1f ms%n", mean);    }}

p99, server A:            29 msp99, server B:            2000 msaverage of the two p99s:  1014 ms (meaningless)true p99, all requests:   2000 mstrue p95, all requests:   30 msmean, all requests:       123.6 ms

Three lessons in six lines. The average of percentiles (1,014 ms) corresponds to no real request. The true p99 shows that 1 in 100 users waits two seconds, while the p95 looks perfectly healthy, so the tail is concentrated. And the mean (124 ms) looks acceptable while 5% of all requests are terrible. This is the opening story: averages hid a sick server. Per-server percentiles on a dashboard, plus correctly aggregated histograms, would have revealed it in minutes.

In a system with many services, a slow request might be slow because of any of them. A trace records one request's journey as a tree of spans: each span is one operation (an HTTP call, a database query, a cache lookup) with a start time, duration, status and attributes, and a parent span.

Trace 4bf92f35...  GET /v1/feed                          total 2,040 ms|-- auth.validate_token                          4 ms|-- feed.read_precomputed (Redis)                6 ms|-- photos.batch_get (PostgreSQL replica)       18 ms|-- recommendations.get                       2,003 ms   <- timeout`-- likes.batch_counts (Redis)                   5 ms

One glance shows where the time went.

For spans from different services to join one trace, each call must carry the trace context: the trace id and the id of the calling span. The W3C Trace Context standard defines a traceparent HTTP header with four fields: a version, a 32-hex-character trace id, a 16-hex-character parent span id, and trace flags, for example 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01 (W3C Trace Context). Every service reads the incoming header, creates child spans, and sends the header on its own outgoing calls. OpenTelemetry's documentation explains this as context propagation (Context propagation).

For asynchronous work, put the trace context into message metadata when publishing to a queue, so the consumer's spans connect to the request that caused them.

Tracing every request at high volume is expensive. Systems sample:

• Head sampling: decide at the start of a request (for example, keep 1%). Cheap, but you may miss the rare slow or failing requests you care about.
• Tail sampling: buffer spans and decide after the request finishes, keeping all errors and slow requests plus a small share of normal ones. More useful, more infrastructure.

OpenTelemetry is a vendor-neutral, open-source set of APIs, SDKs and a collector for producing and exporting traces, metrics and logs. You instrument code once and send the data to whichever backend you use: open-source tools (for example Prometheus for metrics, Jaeger for traces, Grafana for dashboards) or a cloud provider's or vendor's observability service. Many frameworks and libraries can be instrumented automatically.

A short Java illustration of manual instrumentation with the OpenTelemetry API (a fragment, not a complete program):

Span span = tracer.spanBuilder("recommendations.get").startSpan();try (Scope scope = span.makeCurrent()) {    span.setAttribute("user.segment", segment);    return recommendationClient.fetch(userId, deadline);   // the client propagates traceparent} catch (TimeoutException e) {    span.setStatus(StatusCode.ERROR, "timeout");    return Recommendations.empty();                          // fallback (article 23)} finally {    span.end();}

Article 02 introduced SLIs (measured indicators), SLOs (targets) and SLAs (contracts), following Google's SRE book (Service Level Objectives). Observability makes them concrete:

Measure SLIs as close to the user as practical (load balancer logs, client telemetry, synthetic checks), because a server cannot report a request that never reached it.

The SRE book's guidance is to alert on symptoms (users are affected: errors are high, latency is high) rather than every possible cause (CPU is high, one replica is lagging), because symptom alerts reflect real impact and cause alerts tend to be noisy (Monitoring Distributed Systems). Causes belong on dashboards for diagnosis.

Good alerts are:

• Actionable: someone can do something now.
• Urgent when they page: non-urgent issues become tickets, not 3 a.m. pages.
• Linked to a runbook with first steps.
• Rare enough to be trusted. Alert fatigue makes people ignore pages.

An error budget (article 24) can be burned fast or slowly. The SRE workbook recommends alerting on how quickly the error budget is being consumed, using combinations of a short and a long window so that a severe problem pages quickly and a slow leak is still caught without flapping (Alerting on SLOs). The intuition: if, at the current error rate, you would use the whole month's budget in two days, page someone now; if you would use it in three weeks, open a ticket.

• Liveness and readiness checks tell orchestrators and load balancers whether to restart a process or send it traffic (article 08). Keep them shallow.
• Synthetic monitoring runs scripted user journeys from outside the system every minute: sign in, load the feed, upload a test photo. It catches problems that internal metrics miss (DNS, certificates, CDN, a broken front end) and measures availability as users experience it.

Queues hide problems because nothing fails visibly; work just waits. For every queue and stream, track:

• depth or consumer lag (messages waiting);
• age of the oldest message (the user-facing delay);
• processing rate and error rate per consumer;
• dead-letter queue size (article 11);
• end-to-end latency from event creation to effect (for example "upload to thumbnail ready"), which is the real SLI for the asynchronous path.

Observability data is not free. A quick estimate for Pixly's API:

3,500 requests/s at peak x 5 log lines per request x 500 bytes per line= 8.75 MB/s, about 750 GB per day at that rate (less on average)

Keeping that for 30 days in a searchable log system is expensive. Control it:

• Log levels and sampling: log every error, sample routine success logs.
• Short retention for verbose logs; longer for metrics (which are compact) and audit logs (which are required).
• Bounded metric cardinality.
• Tail-based trace sampling that keeps the interesting requests.

Replaying the opening story: the per-instance p99 dashboard shows one server at 2 seconds; its traces show slow network spans; it is drained and replaced within minutes, and the feed latency SLO alert, not social media, is what told the team.

• Dashboards of averages.
• Paging on CPU, disk and every internal metric.
• No request or trace ids, so logs cannot be correlated.
• Not measuring the asynchronous path's end-to-end delay.
• Logging tokens, passwords or full personal details.
• Treating observability as an afterthought added after launch.

• In the Harden phase, give each main flow an SLI and an SLO.
• Say "golden signals per service, latency as percentiles from histograms".
• Mention distributed tracing with context propagated through calls and queues.
• Name two or three alerts and why they are symptoms.
• For async pipelines, name lag or oldest-message age as the key metric.

• Next: Monolith vs Modular Monolith vs Microservices: How to Choose starts Module 8; observability is one of the deciding costs there.
• Later: every case study ends its Harden phase with an observability section built from this article.