06 - Sampling and Model Behaviour¶
LLM output is not produced like a deterministic function such as calculateTax(amount). The model predicts a probability distribution over possible next tokens and a decoding strategy selects from that distribution.
That means two identical requests can produce different valid answers.
The core mental model¶
At each generation step the model produces probabilities for possible next tokens.
Simplified example:
Input: "The capital of France is"
Possible next tokens:
Paris 0.96
Lyon 0.01
Marseille 0.005
...
For obvious facts, one option may dominate. For creative or ambiguous tasks, several continuations may be plausible.
Temperature¶
Temperature changes how sharply or broadly the model samples from its token probabilities.
Conceptually:
- lower temperature -> more conservative, repeatable choices,
- higher temperature -> more diverse, exploratory choices.
Example use cases:
Low temperature:
- classification
- extraction
- code transformation
- structured business output
Higher temperature:
- brainstorming
- naming ideas
- creative writing
- exploring alternatives
Temperature does not make the model smarter. A high value does not improve reasoning quality; it mostly increases diversity and randomness.
Top-p¶
Top-p, also called nucleus sampling, limits candidate tokens to the smallest set whose combined probability reaches a threshold.
You usually do not need to tune both temperature and top-p aggressively. In application engineering, defaults are often better until evaluation shows a specific need.
Determinism is relative¶
Even with conservative sampling settings, full determinism may not be guaranteed across:
- different model versions,
- provider backend changes,
- floating-point execution differences,
- reasoning systems,
- tool usage,
- changing retrieved context.
If a business rule must always behave exactly the same, implement it in deterministic code.
Bad design:
Prompt:
"If total > 10,000 EUR, require manager approval."
Better design:
requires_approval = total > 10_000
The model can explain the rule, but application code should enforce it.
Seed¶
Some APIs expose a seed to make sampling more reproducible. This can help tests, but it should not be treated as a universal guarantee of identical output forever.
Model upgrades or infrastructure changes may still alter results.
Why repeated answers differ¶
Suppose the task is:
Give me three names for an AI monitoring product.
There is no single correct output. Many token paths are valid, so repeated calls naturally produce different answers.
By contrast:
Extract invoice_number from this document.
should be engineered toward low variance through clear instructions, structured output, validation, and evaluation.
Reasoning behaviour¶
Reasoning-capable models may spend additional computation before producing an answer. This can improve performance on planning, coding, mathematics, and complex decision tasks, but does not remove uncertainty.
Important distinction:
more reasoning
!=
guaranteed correctness
Reasoning output should still be validated when it affects application state.
Task type should influence model behaviour¶
A useful application-level classification is:
| Task | Desired behaviour |
|---|---|
| Extraction | Stable and constrained |
| Classification | Stable and constrained |
| Code generation | Mostly constrained, some flexibility |
| Planning | Exploratory but grounded |
| Brainstorming | Diverse |
| User-facing prose | Natural, moderately flexible |
This is more useful than trying to find one global sampling configuration for the entire application.
Variance and evaluation¶
Because outputs can vary, testing one example once is weak evidence.
Instead of:
Prompt v2 gave a better answer once.
prefer:
Run evaluation set
|
+--> quality score
+--> failure rate
+--> format compliance
+--> latency
+--> cost
For some tasks, multiple runs per test case may be useful to measure variance.
Common mistakes¶
Mistake 1: using temperature as a quality knob¶
Increasing temperature does not fix weak reasoning or missing context.
Mistake 2: demanding exact textual equality¶
Two semantically equivalent answers may use different wording. Exact string tests are often inappropriate for open-ended generation.
Mistake 3: placing deterministic rules in prompts¶
If something can be represented safely as code, validation, or schema, do that.
Mistake 4: assuming reasoning models cannot hallucinate¶
Stronger reasoning reduces some failures, but models can still make incorrect assumptions, misread context, or invent facts.
Developer takeaway¶
Treat model output as a probabilistic component inside a deterministic software system.
Use model flexibility where it creates value, and surround it with deterministic boundaries where correctness matters.