Skip to Content

Agent Observability (Tracing and Eval)

When to use

See what your agent actually did — every model call, every tool invocation, every state transition — and find regressions fast.

Analogy

Black box flight recorder for agents. Every message, every tool call, every retry is stamped with timestamps and logged to a dashboard.

Data-flow diagram

   agent.run -> [trace span] model + tool + tool + model
                        |
                        v
              LangSmith / Langfuse dashboard
              + eval suite re-runs over golden traces

Deep explanation

Agent observability platforms capture every LLM call, tool call, and state transition with structured metadata. They let you replay a session, find the prompt that caused the bad output, and run regression evals on historical runs. Use LangSmith (deep LangChain integration) or Langfuse (open source, multi-provider). Always add tracing in dev; always add it in prod.

Examples

Example 1

import langsmith
from langchain_openai import ChatOpenAI
langsmith.trace(enabled=True, project='agent-prod')
chain = prompt | ChatOpenAI() | parser
print(chain.invoke({'topic':'cats'}))   # automatic trace capture

LangSmith auto-traces LangChain Runnables; minimal code.

Example 2

# Langfuse (provider-agnostic)
from langfuse.openai import OpenAI
oai = OpenAI(api_key='sk-...', base_url='https://...')
resp = oai.chat.completions.create(model='gpt-4o-mini',
    messages=[{'role':'user','content':'Hello'}])

Langfuse.openai SDK wraps OpenAI and adds tracing; works for any provider.

Example 3

# run the eval suite on golden traces
from langsmith.evaluation import evaluate
evaluate(chain.invoke, data='golden-set-2026-05', evaluators=[length, json_valid])

evaluate() replays a golden dataset through your chain; surface regressions early.

Common mistake

Tracing only in dev. If you cannot see production runs, you cannot debug production.

Key takeaway

Add tracing from day one. Use LangSmith for LangChain-heavy code; Langfuse for cross-provider. Replay golden datasets on every release.

Production Failure Playbook

Failure scenario 1: no-tracing-on-outage

Failure scenario 2: trace-explosion