Skip to Content

Temperature and Top-P Sampling

When to use

Set how deterministic or creative the model is — choose based on whether you need a single correct answer or many varied suggestions.

Analogy

Temperature is the model’s creativity dial. 0 is a calculator; 1 is an artist. Top-p cuts off the long tail of low-probability tokens.

Data-flow diagram

   temperature = 0   -> greedy / deterministic (good for extraction)
   temperature = 0.7 -> balance of variety and cohesion (paraphrase)
   temperature = 1.0 -> high variety (brainstorming)

   top_p = 0.9 is a common default

Deep explanation

Temperature scales the logit distribution before sampling: 0 picks the highest-probability token (deterministic); 1 samples proportional to probability (more variance). Top-p (nucleus sampling) cuts off the lowest-probability tokens whose cumulative probability is outside the top-p mass. Use temperature=0 for tasks with a single correct answer (extraction, classification). Use temperature 0.7 to 0.9 for creative or paraphrase tasks.

Examples

Example 1

resp = openai.chat.completions.create(
    model='gpt-4o-mini', temperature=0,
    messages=[{'role':'user','content':'Extract the city from: "I live in Berlin."'}])

temperature=0 plus top_p=1 gives deterministic best-of-classification answer.

Example 2

resp = openai.chat.completions.create(
    model='gpt-4o-mini', temperature=0.7, top_p=0.9,
    messages=[`{'role':'user','content':'Suggest 5 product names for an AI assistant for dentists.'}`])

temperature=0.7 plus top_p=0.9 is the production default for natural copy.

Example 3

responses = [openai.chat.completions.create(
                model='gpt-4o-mini', temperature=1.0,
                messages=[{'role':'user','content':'Invent a brand new fruit.'}])
               for _ in range(5)]
for r in responses:
    print('-', r.choices[0].message.content)

temperature=1.0 is for brainstorming where diversity matters more than coherence.

Common mistake

Using temperature=1.0 for extraction with stable answers expected. Conversely, using temperature=0 kills variety for creative writing.

Key takeaway

temperature=0 for factual/structured tasks; 0.7 to 0.9 for natural-sounding prose; 1.0 for brainstorming. Top-p 0.9 is a sensible default.

Production Failure Playbook

Failure scenario 1: extraction-uses-high-temperature

Failure scenario 2: top-p-too-narrow