Tools change every year. Fundamentals compound for decades. The strongest candidates can explain how LLMs behave, not just how to call them.
What? Temperature (0..1) reshapes the probability distribution the LLM samples from. At temp=0 the model is deterministic (greedy). At temp=1 the distribution is preserved; the LLM is “creative.”
Why? Because the wrong temperature breaks reproducibility, accuracy, or style — sometimes all three.
How? Mental model:
Scenario 1 — Intent-classification routing
You route customer messages to agents based on intent. Same input must produce same route. Pick: temperature=0 (or 0.0). Tests must be reproducible; non-deterministic routing breaks both A/B and compliance auditing.
Scenario 2 — Conversational agent explaining fees
The user asked a question; response should be clear, helpful, with slight conversational variation. Pick: temperature=0.4. Too low = robotic. Too high = hallucinated details.
Scenario 3 — Brainstorming product names
The user wants creative ideas. Pick: temperature=0.9 with top_p=0.95. Somewhat random, somewhat confident.
What? top_p (nucleus sampling) keeps the smallest set of tokens whose cumulative probability is ≥ top_p. The model samples only from that set. Temperature reshapes the entire distribution.
Why? Because they interact. Using both is not the same as using either alone.
How?
Scenario 1 — Code generation
Pick: temperature=0.2, top_p=0.95. Slight variation is OK; pool should still be the high-probability tokens. Don’t allow alt-syntax branching.
Scenario 2 — Legal contract clause rewrite
Pick: temperature=0.0, top_p=1.0. The text must be deterministic. Same input = same output for compliance.
Scenario 3 — Poetry / creative narrative
Pick: temperature=0.85, top_p=0.99. High variation is desirable; almost all tokens are eligible.
What? Most production LLM endpoints accept a response_format={”type”: “json_object”} flag (OpenAI, Groq, Anthropic) that guarantees the response is valid JSON. Without it, the LLM is free to wrap answers in markdown, prose, or apologies.
Why? Because downstream systems need to parse the result. JSON is the lingua franca of APIs; prose is not.
How? Four reasons to choose JSON:
Scenario 1 — Extracting customer email/intent
Without JSON, the LLM may say “Sure! The customer’s email is john@bank.com and the intent is loan_inquiry.” You’d regex-parse that, fragile. JSON: {”email”: “john@bank.com”, ”intent”: “loan_inquiry”}.
Scenario 2 — Tool calling
Tool APIs expect structured args. “Send an email to bob” → JSON: {”to”: “bob@bank.com”, ”subject”: ”…”, ”body”: ”…”}. The LLM actively populates these.
Scenario 3 — Compliance audit
Auditors want a structured trail: every response carries which rule was triggered and why. JSON: {"is_compliant": false, "violations": ["used 'guaranteed' phrase"]}.
What? A multi-pronged defence, since LLMs occasionally return prose around the JSON, malformed JSON, or invalid types against the schema.
Why? Because a malformed response means the request fails. Production can’t tolerate fail rates above ~0.1%.
How? Layers:
response_format={”type”: “json_object”} (model-native guarantee).\n trim before parsing).Scenario 1: LLM returns prose with a JSON code block embedded.
Fix: regex-strip everything outside the first { and last }. Pydantic validates.
Scenario 2 — LLM returns {”intent”: null}
Fix: Pydantic schema declares intent: Literal[...]; null fails validation. Return a fallback / retry with a stricter prompt.
Scenario 3 — LLM returns a JSON object missing required fields
Fix: Pydantic raises ValidationError. Catch it. Insert a retry prompt: “The previous response was missing confidence. Please re-emit with all fields.”
What? Techniques for getting more reliable answers: zero-shot, few-shot, chain-of-thought, self-consistency, role-prompt, “Answer ONLY using” grounding.
Why? Because prompt design is the most leveraged dial you control. A 200ms prompt rewrite can outperform a $50k re-training.
How? Common techniques:
Scenario 1 — Customer intent classification Pick: few-shot with 5 examples per class. Empty examples often produce hallucinations; 2-3 examples per class stabilise.
Scenario 2 — Math/reasoning problem Pick: chain-of-thought (“Let’s think step by step”). For direct answers, LLMs skip intermediate steps and fail. CoT lets the model verify its arithmetic.
Scenario 3 — Domain-tuned agent (FCA compliance) Pick: role + grounding. “You are a UK FCA compliance officer. Use ONLY the context below. Answer in plain English. If you don’t know, say so.”