From many monitoring tools to one view of what customers experience

The situation

A telecom operator ran customer-facing digital services (self-care apps, online ordering, payments and activation) on top of a mix of modern microservices and long-lived back-office systems. Monitoring had grown team by team. Each group had its own agents, dashboards and alert rules, and the same incident lit up several tools at once without any of them saying whether customers were affected.

The pattern on bad days was familiar: complaints arrived through support channels first, several teams investigated in parallel from their own dashboards, and the root cause sat at a boundary nobody's tool covered.

What had to be true

• One vendor-neutral telemetry pipeline, so instrumentation is done once and backends can change.
• Traces across boundaries, from the app request through the API layer to the back-office call.
• Objectives tied to customer journeys, agreed with the people who own those journeys.
• Alerts that mean action: page when users are being hurt, ticket when a budget is slowly eroding, and nothing else.
• Cost under control, with sampling and cardinality managed centrally rather than service by service.

Architecture

The core was an OpenTelemetry Collector gateway: services and node-level agents send OTLP to a central endpoint per cluster or region, where processing, sampling and export happen once¹. Deployment patterns are compared in OpenTelemetry Collector deployment patterns.

For older services that could not be changed quickly, OpenTelemetry eBPF Instrumentation (OBI) captured spans and rate, error and duration metrics for HTTP and gRPC traffic without code changes². It cannot see business context, so revenue-critical services also received SDK instrumentation for custom attributes. More in OpenTelemetry eBPF instrumentation.

The data flow, end to end

Prometheus held the SLI time series and evaluated recording and alerting rules, which suits numeric operational data well, though not anything that needs exact per-request accounting³.

The hard part: deciding what to promise

The pipeline was engineering. The objectives were negotiation. Each journey (sign in, view bill, pay, activate) needed an indicator measured close to the user, a target the business could defend, and an owner who accepted the consequences of the error budget⁵. In practice that meant:

• Choosing indicators at the edge of the journey, not on internal components that customers never see.
• Starting with targets the service already met most of the time, then tightening, rather than declaring aspirational numbers nobody could hold.
• Writing a short error budget policy per journey that said what happens when the budget runs low.
• Retiring old component alerts deliberately, with each team agreeing what replaced them.

Alerting followed the multiwindow, multi-burn-rate approach in the Google SRE Workbook, which pairs a long window for significance with a short window for fast reset⁴. The practical version for smaller teams is in SLOs and error budgets for small teams.

How the engagement ran

Key decisions and trade-offs

What I'd do again, and differently

Again: trace one journey end to end before instrumenting everything, and make product owners co-own the targets. Differently: set attribute naming and cardinality rules before the first service onboards. Cleaning up inconsistent labels later cost more than agreeing them early.

If you're planning something similar, the fractional CTO page describes how I lead platform and reliability work, or analyse your challenge and bring the result to a call.