hard general · part of Practice Questions · Senior SWE Roadmap · topic form: Observability
Requirements to clarify
- Functional: services emit metrics (counters, gauges, histograms), the system stores them, supports querying/dashboards, and fires alerts when thresholds are breached.
- Non-functional: extremely high write volume (every service, every host, every few seconds), queries need to be fast for dashboards, alerts need to fire within seconds of a real incident.
Core components
- Metric ingestion: agents on each host/service push (or are scraped for) metrics into a collection pipeline, usually buffered through a message queue to absorb bursts.
- Time-series storage: a database optimized for time-ordered, append-heavy writes and range/aggregation queries (downsampling older data to save space — full resolution for recent data, coarser rollups for history).
- Aggregation: pre-aggregating (sum/avg/percentiles) at ingestion or query time — raw per-event storage doesn’t scale for dashboards showing months of history.
- Alerting engine: continuously evaluates rules against incoming metrics (or recent aggregates) and triggers notifications when thresholds are crossed, with debouncing to avoid alert flapping.
Key tradeoffs
- Storage cost vs granularity: keeping full-resolution data forever is prohibitively expensive — rollup/downsampling policies (e.g., full res for 24h, 1-minute rollups for 30 days, hourly beyond that) are the standard answer.