Why rerun an eval on production traces that look bad?

Category: Observability & Tracing

Answer

Production traces are free golden data. If a user complained that the agent gave a bad answer, the trace you already logged IS the input to a regression test. Run your eval suite over a filter “traces where user clicked thumbs-down” — wait, you have those — and you get a reproducible test for every reported bug.

Concrete examples from the fca project context

Example 1

Storage: every trace kept 90 days with metadata (route, user_id, llm_judge_score).

Example 2

Filter: traces where thumbs_down was clicked -> dump to eval suite.

Example 3

Run the eval automatically on the dump; new failing tests = discovered regressions.

Example 4

This becomes a CI gate.

Data flow / flow chart

prod trace {user, llm_call, score} -> eval pipeline -> regression test
  (the bug the user saw is reproducible in CI)

Takeaway

Production traces are gold. Mine them for evals; you’ll close regressions faster.