Why use OpenTelemetry for LLMs
Proprietary observability tools (LangSmith, Helicone, Braintrust) are convenient but create vendor lock-in. OpenTelemetry is an open standard supported by all major observability backends — Jaeger, Grafana Tempo, Honeycomb, DataDog, New Relic. Instrument once, export anywhere.
The OpenTelemetry GenAI Semantic Conventions (released in 2024) define standard attribute names for LLMLLMLarge Language Model — a neural network trained on vast amounts of text data that can understand and generate human-quality language across a wide range of tasks.Learn more → spans: `gen_ai.system`, `gen_ai.request.model`, `gen_ai.usage.input_tokens`, `gen_ai.usage.output_tokens`. Using these conventions makes your traces compatible with any GenAI-aware observability tool.
Set up the OTel SDK
Install: `pip install opentelemetry-sdk opentelemetry-exporter-otlp opentelemetry-instrumentation-openai`. Configure the tracer provider and set up the OTLP exporter: `from opentelemetry import trace; from opentelemetry.sdk.trace import TracerProvider; from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter; provider = TracerProvider(); provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter(endpoint='http://localhost:4318/v1/traces'))); trace.set_tracer_provider(provider)`.
For automatic OpenAI instrumentation: `from opentelemetry.instrumentation.openai import OpenAIInstrumentor; OpenAIInstrumentor().instrument()`. Every OpenAI API call now automatically creates a span with the request model, tokenTokenThe basic unit of text that an LLM processes — roughly corresponding to a word or word-piece. Models read input and produce output in tokens, which is also how API usage is measured and billed.Learn more → usage, and response.
Create custom spans
Wrap logical operations in spans: `tracer = trace.get_tracer('my-app'); with tracer.start_as_current_span('retrieve-context') as span: results = vector_store.search(query); span.set_attribute('retrieved_chunks', len(results)); span.set_attribute('query_length', len(query))`.
Add LLM-specific attributes to your spans: `span.set_attribute('gen_ai.request.model', model); span.set_attribute('gen_ai.usage.input_tokens', usage.input); span.set_attribute('gen_ai.usage.output_tokens', usage.output); span.set_attribute('gen_ai.usage.cost_usd', cost)`. These show up in dashboards as queryable dimensions.
Export and visualise
Run Jaeger locally for development: `docker run -p 16686:16686 -p 4318:4318 jaegertracing/all-in-one`. Open `http://localhost:16686` to browse traces. For production, use a managed OTLP endpoint (Grafana Cloud, Honeycomb, DataDog OTLP endpoint) — just change the exporter URL.
Build Grafana dashboards over your OTel data. Key metrics to visualise: P50/P90/P99 latencyLatencyThe delay between sending a request to an LLM and receiving the first token of the response, often measured as Time to First Token (TTFT).Learn more → by model, token usage over time, error rate by model and endpoint, cost per feature/user. Grafana's OTel integration generates these dashboards automatically from your traces.