Back to Blog
2026-05-15LLM EngineeringObservabilityAWS

Observability in LLM Engineering — What Production Actually Looks Like

Traditional observability tooling covers latency, error rate, and throughput. LLM systems add a new dimension: the quality of reasoning itself. Here is how to instrument it.

The Gap in Traditional Observability

When I spoke on this topic at AWS Summit, I opened with a simple question to the audience: How many of you know your p99 latency for your LLM calls right now?

A few hands. Then: How many of you know the quality distribution of responses your model is producing today?

Almost none.

This is the gap. Traditional APM tools cover infrastructure. LLM observability must also cover semantic quality — and that requires a different approach.

The Three Layers of LLM Observability

Layer 1 — Infrastructure Signals (the table stakes)

  • Token consumption per request (input + output)
  • Latency percentiles (p50, p95, p99) — broken down by model
  • Error rates by error type (rate limits, context overflow, timeouts)
  • Cost per call and cost per user session

These map directly to standard APM and can be captured at the SDK or proxy layer.

Layer 2 — Behavioural Signals (the new territory)

  • Tool call frequency and tool call failure rate
  • Retry and fallback rates within agent loops
  • Context window utilisation (% of max tokens used)
  • Conversation depth before task completion or abandonment

These tell you how the model is behaving, not just whether it responded.

Layer 3 — Quality Signals (the hard part)

  • LLM-as-judge scoring on sampled outputs
  • User correction rate (proxy for semantic accuracy)
  • Task completion rate for agentic workflows
  • Hallucination flag rate via retrieval grounding checks

Layer 3 cannot be captured passively. It requires explicit instrumentation, usually a secondary LLM call scoring the primary one on a rubric you define for your domain.

Implementation Approach on AWS

On AWS, the stack I recommend for teams starting out:

  1. Amazon Bedrock — model invocations with built-in token logging via CloudTrail
  2. Amazon CloudWatch — custom metrics namespace for your LLM signals
  3. AWS X-Ray — distributed tracing across your agent's tool calls
  4. OpenTelemetry SDK — for portable instrumentation that is not AWS-specific

For quality signals, use a lightweight judge model (Haiku-class) running asynchronously against sampled production traffic. Keep the judge prompt simple and specific to your use case.

The Takeaway

If you are running LLMs in production without Layer 2 and 3 observability, you are operating blind. You will know when something breaks, but not why, and not how long it was quietly degrading before it broke.

Start with Layer 1 this week. Add Layer 2 signals as your system matures. Build Layer 3 before you scale.