The first question I ask about any AI system in production is simple: for yesterday's traffic, what did each answer cost, how long did it take, and which ones went wrong? Most teams can't answer it. They have application logs and a monthly invoice from a model provider, and nothing in between.
That gap is closing. OpenTelemetry, the open standard most teams already use for traces and metrics, has semantic conventions for generative AI. They define how to record a model call, its token usage and the tools an agent invokes, so any compatible backend can show it1.
What the conventions give you
The OpenTelemetry blog walks through a real example. A trace from an agent shows a top-level invoke_agent span, child chat spans for each model call, and execute_tool spans for each tool invocation1.
Each span carries standard attributes, including the model requested, input and output token counts, and the reasons the model stopped generating12. There are matching metrics conventions, so you can chart usage and latency without parsing logs3, and separate conventions for agent and framework spans4.
Why this matters more than it sounds
| Question | What answers it |
|---|---|
| Which feature is driving the bill? | Token counts per span, grouped by service and route |
| Why is this answer slow? | The span tree: model time versus tool time versus retrieval |
| Why did the agent loop? | Repeated tool spans and finish reasons inside one trace |
| Did the new model make things worse? | Latency and error metrics split by model, before and after |
Because it is the same OpenTelemetry you use elsewhere, an AI call shows up inside the request that triggered it. You see the database query, the retrieval step and the model call on one timeline, instead of three tools that never agree.
Privacy: metadata yes, content by exception
This is the part I am strict about. Prompts and completions can contain personal data, health information or confidential documents. The OpenTelemetry walkthrough notes that by default no prompt content or tool arguments are captured, only metadata such as model names, token counts and durations. Content capture is an explicit opt-in1.
Keep it that way in production. Turn on content capture only in development, in a sandbox with synthetic data, or behind redaction at an AI gateway. A trace store full of raw prompts is a data breach waiting for a misconfigured dashboard.
This lines up with how I design governed AI platforms generally, which I describe in securing MCP servers and UAE PDPL and AI.
A caution on stability
The GenAI conventions are still marked as in development on the official pages, and existing instrumentations can opt in to the newest experimental version with an environment variable2. Attribute names have already changed between versions; one commentary notes, for example, a move from gen_ai.system to gen_ai.provider.name5.
That is not a reason to wait. It is a reason to pin the version you emit, keep dashboards tolerant of renamed attributes, and plan a small migration when the conventions settle.
How I set it up
- Instrument at the gateway. If every model call passes through one AI gateway, instrument it once and every product inherits the telemetry.
- Add agent and tool spans in the orchestration layer, so a multi-step task appears as one trace rather than a pile of unrelated calls.
- Tag with business context: tenant, feature and use case. Cost per feature is the number leadership actually asks for.
- Metadata only in production. Content capture off, redaction on, retention short.
- Build three views: cost per feature per day, latency by step, and failures by finish reason and tool.
- Connect to evaluation. Link trace IDs to your evaluation runs so a quality regression can be traced to a specific prompt or model change. The RAG production checklist covers the evaluation side.
How I can help
I design AI platforms where cost, latency and quality are visible from day one: gateways, tracing, evaluation and governance. See my AI architecture work, or analyse your challenge and bring the result to a call.
- 1Inside the LLM Call: GenAI Observability with OpenTelemetryOpenTelemetry blog · 14 May 2026
- 2Semantic conventions for generative client AI spansOpenTelemetry
- 3Semantic conventions for generative AI metricsOpenTelemetry
- 4
- 5OpenTelemetry GenAI semantic conventionsGreptime · May 2026