What’s a reasonable per-request LLM cost and how do you control it?

Category: LLM & Prompt Engineering

Answer

Track tokens in metrics (input + output), tag by route + tenant. Per-request cost = (tokens_in * $in/M + tokens_out * $out/M). Set a token-per-tenant budget and a circuit breaker on bursts. Cache repeats (semantic or exact). For chat, expect $0.005-0.05 per request on gpt-4o-mini.

Concrete examples from the fca project context

Example 1

Counter: llm_tokens_total{route, model} -> $/month dashboard.

Example 2

Semantic cache: same embedding -> return prior response.

Example 3

Snapshot-and-route: simple queries go to gpt-4o-mini; complex ones to gpt-4o.

Example 4

Per-tenant cap: if p99 tokens/min > 10x median, rate-limit.

Data flow / flow chart

pre-flight token estimate -> route to cheap model for routine
heavy / uncertain -> expensive model
cache + tenant cap keep the bill flat

Takeaway

Tokens in metrics, semantic cache, route by complexity. Three controls keep the bill flat.