Skip to content
All case studies

Case study 072022

Observability Platform

Unified tracing, metrics and SLO alerting across 40+ services, so pages mean something again.

Role
Senior engineer
Team
Platform team of 4
Timeline
6 months
Focus
Platform

−70%

pages per week

−55%

mean time to resolve

40+

services instrumented

Services emit OpenTelemetry signals through one collector; deploy and flag events are overlaid so incidents line up on one timeline.40+ services to OTel collector; Deploys & flags to OTel collector (annotations); OTel collector to Traces; OTel collector to Metrics; OTel collector to Logs; Metrics to SLO alerts40+ servicesauto-instrumentedDeploys & flagschange eventsOTel collectorsampling · routingTracesMetricsPrometheusLogsSLO alertsmulti-window burn rateannotations
Services emit OpenTelemetry signals through one collector; deploy and flag events are overlaid so incidents line up on one timeline.

The problem

Every team had its own logging, there was no distributed tracing, and alerts fired on CPU spikes nobody cared about. Real incidents took hours to localise across service boundaries.

What I owned

  • OpenTelemetry auto-instrumentation shipped through the shared service template.
  • SLO definitions with product owners, and multi-window burn-rate alerting.
  • Incident dashboards that line up traces, deploys and feature flags on one timeline.

Key decisions

  1. 1

    Instrument through the template

    Tracing came for free with the shared service template, so adoption didn't depend on 40 teams finding the time.

  2. 2

    Alert on user pain, not machine stats

    SLOs on latency and error rate replaced threshold alerts. Burn-rate windows catch fast outages and slow leaks without the noise.

Outcome

Pages dropped from about 120 a week to under 40, and the ones left are real. Incidents are localised in minutes from a single dashboard.

  • OpenTelemetry
  • Prometheus
  • Grafana
  • Kubernetes
  • Go