How do you add a custom business metric alongside the defaults?

Category: Observability & Tracing

Answer

Use prometheus_client.Counter, Histogram, Gauge directly, or expose them through a registry the Instrumentator already scraped. Increment in your service: RETRIEVAL_HITS.labels(source="faq").inc() reads like business intent. Plot alongside HTTP metrics and act when it shifts.

Concrete examples from the fca project context

Example 1

Counter(“retrieval_hits_total”, “RAG retrieval cache hits”, [“source”]).inc().

Example 2

Histogram(“llm_latency_seconds”, “LLM latency per call”, buckets=[.05,.1,.25,.5,1,2,5]).

Example 3

Gauge(“agent_pending”, “Active agent runs”) updated by an asyncio task.

Example 4

Custom metrics live in the same /metrics endpoint — Grafana picks them up.

Data flow / flow chart

service -> inc(retrieval_hits.labels(source="faq").inc())
  Prometheus scrapes -> Grafana bar over time

Takeaway

Counter for “how many”, Histogram for “how long”, Gauge for “how alive”. Memorize the three.