How do you budget an LLM’s context window?

Category: LLM & Prompt Engineering

Answer

Priority order: (1) system prompt, (2) the immediate question + retrieved top-K, (3) recent conversation, (4) older turns. Trimming strategy: keep recent N turns verbatim, summarize older turns. Recency bias is real; the model gives more weight to the last 2-4K tokens.

Concrete examples from the fca project context

Example 1

Compose: system (200 tok) + top-K (1500 tok @ 300 each) + last_5_turns (~2000 tok) = ~3700 tok.

Example 2

Trim: when total > budget, summarize older turns into a single “summary_of_earlier” message.

Example 3

Drop: extraneous tool outputs that the answer doesn’t reference.

Example 4

Always pre-compute token estimates (o200k_base) before sending.

Data flow / flow chart

system  200
top-K  1500
recent 2000
---
total ~4000 tokens (gpt-4o has 128K but quality drops past ~32K)

Takeaway

Budget the context yourself. Long prompts degrade quality; trim and summarize aggressively.