Skip to Content

Prompt Caching and Semantic Caches

When to use

Reduce LLM cost and latency by reusing previous answers — either via exact-text caching or by semantic similarity.

Analogy

Librarian keeping a frequently-asked-questions list at the desk; saves work and is fast.

Data-flow diagram

   Prompt cache (provider side): cache prefix based on byte-identity
   Semantic cache (app side): vector index, 'is this close enough to a prior query?'

Deep explanation

Prompt caching (provider-side) deflates cost when the same prefix is reused. Semantic caches (app-side) dedupe by meaning: ‘how do I reset my password?’ and ‘I cannot log in’ should ideally hit the same cached answer. Combine both. Risks: stale answers and semantic-false positives; always version the cache.

Examples

Example 1

resp = openai.chat.completions.create(
    model='gpt-4o-mini',
    messages=[{'role':'system','content': system_prompt,
               'cache_control': {'type':'ephemeral'}},
              {'role':'user','content': user_q}])

Provider-side caching is automatic when the prefix is byte-identical.

Example 2

from sentence_transformers import SentenceTransformer
import numpy as np
m = SentenceTransformer('all-MiniLM-L6-v2')
recent = []  # list of (embedding, answer)
def cached_answer(q, threshold=0.92):
    e = m.encode(q)
    if not recent: return None
    sims = [(np.dot(e, ee) / (np.linalg.norm(e)*np.linalg.norm(ee)), a)
            for ee, a in recent]
    best = max(sims, key=lambda x: x[0])
    return best[1] if best[0] >= threshold else None

Application-side semantic cache dedupes by meaning; threshold 0.92 is a sensible default.

Example 3

def answer(q):
    cached = cached_answer(q)
    if cached: return cached
    fresh = call_llm(q)
    if freshness_score(fresh) > 0.7:
        recent.append((m.encode(q), fresh))
    return fresh

Quality-check answers before inserting into a semantic cache; never cache hallucinated responses.

Common mistake

Caching every answer forever — the underlying world changes; answers go stale. Also setting similarity threshold too low.

Key takeaway

Provider-side caching for static prefixes. Semantic cache for high-volume FAQs. Version the cache and invalidate aggressively.

Production Failure Playbook

Failure scenario 1: stale-cache-after-model-update

Failure scenario 2: cache-threshold-too-low