Observability

must-know senior · part of Skills & Topics · Senior SWE Roadmap · related: Message Queues & Event-Driven Architecture · Microservices vs Monolith · Scalability Fundamentals · question form: Design a Metrics Monitoring & Alerting System

Why this is a senior-level topic

Anyone can design a system that works when everything is healthy. A senior-level answer proactively addresses “how would you know if this is broken in production, and how would you find out why, in minutes rather than hours?” — before the interviewer even asks. Observability is the set of tools and practices that make a running distributed system’s internal state inferable from its external outputs (logs, metrics, traces). It’s not “monitoring” in the older, narrower sense of “is the server up” — it’s being able to answer novel questions about a system’s behavior you didn’t anticipate when you instrumented it.

The three pillars

Pillar	Captures	Good for	Bad for	Relative cost
Logging	Discrete, timestamped events with arbitrary detail	Root-causing one specific failure — exact error message, stack trace, request payload	Trends over time; correlating across services without extra tooling	Highest — especially unstructured, unsampled logs at scale
Metrics	Numeric time series, pre-aggregated (counters, gauges, histograms)	Dashboards, trend detection, alerting thresholds, capacity planning	Explaining why — a spike tells you that something’s wrong, not which request or user	Cheap — fixed cardinality, compact storage
Tracing	The path and timing of one request across every service it touches	Finding the slow or failing hop in a multi-service call chain	Aggregate trends — usually sampled, not every request, so it’s not a substitute for metrics	Moderate — grows with (request volume × services touched), mitigated by sampling

A few things worth being precise about:

• Structured logging (JSON with consistent fields) beats free-text logging once you have more than a handful of services — it lets you query “all logs where user_id=42 and level=error” instead of grepping.
• Metric cardinality is a real constraint: a Prometheus-style label with unbounded values (e.g., user_id as a label) can blow up storage and query cost — cardinality explosions are a classic production incident. High-cardinality dimensions belong in logs or traces, not metric labels.
• The three pillars aren’t independent — the highest-leverage move is correlating them with a shared identifier (trace/request ID) so you can pivot from a metric spike to the specific traces that caused it, to the exact log lines for one of those traces.

flowchart TD
    Req["Incoming request<br/>request_id = abc-123"] --> Log["Log line<br/>ts, level, request_id=abc-123, msg"]
    Req --> Metric["Metric point<br/>http_requests_total{route=/checkout} += 1"]
    Req --> Trace["Trace span<br/>trace_id=abc-123, span=checkout, duration=340ms"]
    Log --> Debug["Incident debugging:<br/>see a latency spike in the metric →<br/>find a slow trace with that shape →<br/>jump to logs for that exact trace_id"]
    Metric --> Debug
    Trace --> Debug

Distributed tracing, worked example

Tracing propagates a trace ID through every hop of a request, and each hop records a span — a named operation with a start time, duration, and a parent span ID. The collection of spans sharing a trace ID forms a tree (a “waterfall”) showing exactly where time was spent.

sequenceDiagram
    participant Client
    participant GW as API Gateway
    participant Ord as Order Service
    participant Pay as Payment Service

    Client->>GW: POST /checkout (no trace context yet)
    Note over GW: generates trace_id=T1<br/>starts span S1 (gateway), 210ms total
    GW->>Ord: POST /orders (header: traceparent=T1-S1)
    Note over Ord: starts span S2, parent=S1
    Ord->>Pay: POST /charge (header: traceparent=T1-S2)
    Note over Pay: starts span S3, parent=S2
    Pay-->>Ord: 200 OK (span S3 ends: 120ms)
    Ord-->>GW: 201 Created (span S2 ends: 180ms)
    GW-->>Client: 201 Created (span S1 ends: 210ms)
    Note over GW,Pay: Trace T1: S1 (210ms) wraps S2 (180ms) wraps S3 (120ms).<br/>S3 (Payment Service) accounts for the bulk of S2's time —<br/>the waterfall view makes Payment Service the obvious first place to look.

The propagation mechanism (the traceparent header, or equivalent) is the part people gloss over and interviewers probe: the trace ID and parent span ID have to be passed along on every outbound call — HTTP headers, gRPC metadata, or message headers for async hops (see Message Queues & Event-Driven Architecture — once work crosses a queue, someone has to deliberately carry the trace context into the message, or the trace breaks at that boundary). OpenTelemetry is the current vendor-neutral standard for this — instrument once, export spans to whichever backend (Jaeger, Tempo, a vendor APM) you choose.

Because tracing every single request at scale is expensive, production systems sample — trace 1% of traffic, or trace 100% of error/slow requests and a small percentage of everything else (“tail-based sampling,” which requires buffering a whole trace before deciding to keep it, since you don’t know a request will be slow until it finishes).

SLI / SLO / SLA — the hierarchy, worked example

These three terms get used loosely; know the precise distinction and be ready to give a concrete number for each.

• SLI (Service Level Indicator): the thing you actually measure. Example: p99 latency of the checkout endpoint, measured this week: 260ms.
• SLO (Service Level Objective): the internal target for that indicator, set with a safety margin below the SLA so you have room to notice and fix a regression before it becomes a contractual breach. Example: p99 latency < 300ms.
• SLA (Service Level Agreement): the external, often contractual commitment to a customer, usually with a consequence (service credits, penalties) if breached. Example: p99 latency < 500ms, 99.9% uptime — or the customer receives service credits.

flowchart TD
    subgraph SLA["SLA — external, contractual<br/>p99 < 500ms · 99.9% uptime<br/>breach ⇒ customer gets service credits"]
        subgraph SLO["SLO — internal target, stricter than the SLA<br/>p99 < 300ms · 99.95% uptime<br/>(buffer so you catch drift before breaching the SLA)"]
            SLI["SLI — the measured indicator, right now<br/>actual p99 this week: 260ms<br/>actual uptime this week: 99.97%"]
        end
    end

The SLO is deliberately tighter than the SLA — it’s the internal early-warning threshold. If you only alerted at the SLA boundary, by the time you got paged you’d already be breaching your contract; the SLO gives engineering room to react first.

Error budgets

An error budget is the flip side of an SLO, expressed as an allowance rather than a target: if the SLO is 99.9% availability over a 30-day window, the error budget is the remaining 0.1% — about 43.2 minutes of downtime that period is allowed to have without breaching the objective.

The senior-level part of this concept is what it’s used for: an error budget is a shared, quantified way to trade off velocity against reliability, instead of arguing about it qualitatively.

flowchart TD
    Start["SLO: 99.9% availability this month<br/>Error budget: 0.1% ≈ 43 min downtime"] --> Track["Track cumulative downtime<br/>consumed so far this period"]
    Track --> Check{"How much budget remains,<br/>and at what burn rate?"}
    Check -->|"Plenty left, low burn rate"| Ship["Ship features at normal/faster pace —<br/>budget signals the system can absorb some risk"]
    Check -->|"Nearly exhausted / burn rate spiking"| Freeze["Freeze non-critical releases,<br/>prioritize reliability work,<br/>only ship fixes until burn rate drops"]
    Check -->|"Budget fully spent"| Postmortem["Mandatory postmortem;<br/>reliability is the top priority<br/>until the budget resets next period"]

If a team is well under budget, that’s a signal they can afford to ship faster and take on more release risk — being too reliable relative to the SLO is itself a (mild) signal of under-investment in velocity. If a team keeps blowing through budget, that’s an objective, pre-agreed trigger to halt feature work and fix reliability — removing the political fight over “can we ship this” by having decided the threshold in advance. This is the concept that separates a senior observability answer from a junior one: it’s not just “we have dashboards,” it’s “we have a policy that converts a dashboard number into a release decision.”

RED method vs USE method

Two complementary frameworks for deciding what to instrument, aimed at different layers of the stack.

Method	Applies to	Signals	Question it answers
RED — Rate, Errors, Duration	Request-driven services (APIs, microservices)	Requests/sec, error rate (%), latency distribution (p50/p95/p99)	“Is this service healthy, from the perspective of whoever calls it?”
USE — Utilization, Saturation, Errors	Resources (CPU, disk, memory, network, connection pools, thread pools, queues)	% time busy, amount of queued/backed-up work, error/failure counts	”Is this resource close to (or already) a bottleneck?”

Use RED at the service boundary — it’s what a caller experiences, and it’s what you’d put on a service’s primary dashboard and alert on. Use USE one layer down, on the infrastructure a service depends on, typically as the next step when RED tells you something is wrong but not why — e.g., RED shows checkout’s p99 latency spiked; USE on the database connection pool shows it pinned at 100% utilization with a deep wait queue — now you know where to look.

Alerting design

The core principle: alert on symptoms that affect users, not on every internal cause. A disk at 85% full isn’t inherently a page — it’s a ticket. Checkout error rate above 5% for five minutes is a page, because users are experiencing it right now.

Violating this principle is the direct cause of alert fatigue: too many low-signal alerts train on-call engineers to ignore or delay-triage pages, which means the one alert that actually matters gets the same lukewarm response as the noise around it. Fixes: alert on symptoms (RED-style, user-facing) rather than every possible cause (USE-style, internal); require alerts to be actionable (if there’s nothing a human can do right now, it shouldn’t page); deduplicate/group related alerts instead of firing one per affected instance; route non-urgent internal signals to a ticket queue or a dashboard, not a pager.

flowchart TD
    Sym["Symptom detected:<br/>checkout error rate > 5% for 5 min<br/>(user-impacting, actionable)"] --> Page["Page primary on-call"]
    Cause["Internal signal:<br/>one pod out of 40 restarted once<br/>(not user-impacting)"] --> Ticket["Log it / dashboard only —<br/>no page"]
    Page --> Ack{"Acknowledged<br/>within 5 minutes?"}
    Ack -->|No| Escalate["Escalate to secondary on-call"]
    Ack -->|Yes| Investigate["Investigate using dashboards → traces → logs"]

Paging philosophy in practice: page only for things that are (a) currently harming users or clearly about to, and (b) something the on-call engineer can act on immediately. Everything else — capacity trending toward a limit next month, a non-critical background job failing once — belongs in a ticket or a daily digest, not a 3am page.

Observability pipeline architecture

The mechanical pieces, end to end: instrumented applications emit logs/metrics/traces, an agent or sidecar collects and forwards them, a central collector batches/samples/routes, and each signal type lands in a purpose-built backend that dashboards and alerting read from.

flowchart LR
    subgraph Hosts["App hosts / containers"]
        App1["App instance<br/>(logs, metrics, traces)"]
        App2["App instance<br/>(logs, metrics, traces)"]
    end
    App1 --> Agent1["Agent / sidecar<br/>(OTel Collector, Fluent Bit)"]
    App2 --> Agent2["Agent / sidecar"]
    Agent1 --> Collector["Central Collector<br/>(batches, samples, routes)"]
    Agent2 --> Collector
    Collector --> MetricsStore[("Metrics store<br/>Prometheus / TSDB")]
    Collector --> LogStore[("Log store<br/>Elasticsearch / Loki")]
    Collector --> TraceStore[("Trace store<br/>Jaeger / Tempo")]
    MetricsStore --> Dash["Dashboards<br/>(Grafana)"]
    LogStore --> Dash
    TraceStore --> Dash
    MetricsStore --> Alert["Alerting<br/>(Alertmanager / PagerDuty)"]
    Alert --> OnCall["On-call engineer paged"]

Pillar	Common open-source	Common managed
Logging	Elastic/ELK stack, Grafana Loki	Datadog Logs, Splunk, CloudWatch Logs
Metrics	Prometheus + Grafana, Graphite	Datadog Metrics, CloudWatch Metrics
Tracing	Jaeger, Zipkin, Grafana Tempo	Datadog APM, Honeycomb, AWS X-Ray
Instrumentation standard	OpenTelemetry — vendor-neutral SDKs + collector, exports to any of the above	—

Interview angles

• Senior-level system design answers proactively ask (and answer) “how would we know if this is broken in production, and how would we debug it fast?” — don’t wait to be asked; mention it when you sketch the architecture.
• “How would you debug a slow request across five microservices?” → distributed tracing, walk through trace ID propagation across sync and async hops, and reading the waterfall to find the slow span.
• “What’s the difference between an SLI, SLO, and SLA?” → give the precise hierarchy with concrete numbers, not just definitions — and explain why the SLO is set tighter than the SLA.
• “How do you decide what to alert on?” → symptom-based (RED, user-facing), not cause-based (USE, internal); actionable; explain how this avoids alert fatigue.
• “What’s an error budget and why does it matter?” → this is the tell for genuine senior-level fluency — connect it to a concrete release-velocity decision, not just a definition.
• “Tracing every request is expensive — what do you do?” → sampling strategy (fixed-rate vs tail-based/error-biased sampling), and the trade-off each implies for what you can debug after the fact.

My Notes