ASAmol Shukla
45 mins lesson duration•9 mins read

Lesson 7: Top-K & Top-P (Nucleus) Sampling

The other sampling dials: top-k trims the candidate list, top-p trims by probability mass. Learn to combine them.

Beyond Temperature: Trimming the Candidate List

Temperature changes the shape of the distribution. Top-k and top-p change which tokens are allowed to be sampled at all. They cut off the "long tail" of absurdly unlikely tokens.

Top-K: "Only the K Most Likely Tokens"

Sort all candidate tokens by probability and keep only the top K. The model samples only from those.

  • top_k = 1 → greedy (always the single best token).
  • top_k = 40 → a common default; trims rare garbage tokens.
  • Great for removing typos, gibberish, and off-topic completions.

Top-P (Nucleus Sampling): "Keep the Mass"

Sort candidates from most to least likely, then keep adding tokens until their cumulative probability reaches p (e.g. 0.9).

  • top_p = 0.9 means: "sample from the smallest set of tokens that covers 90% of the probability."
  • Adapts automatically: in an easy, confident spot the set is tiny; in an uncertain spot it grows.
Setting Meaning Typical Value
top_k Max candidates considered 40–100 (or off)
top_p Probability mass covered 0.9–0.95
temperature Distribution sharpness 0–1 (task-dependent)

How They Interact

APIs apply top-k, then top-p, then temperature, then sample. The practical recipes:

  • Deterministic: temperature 0 (top-k/top-p irrelevant).
  • Creative but sane: temperature 0.9–1.1, top-p 0.9.
  • Precise: temperature 0.2, top-p 0.8.
  • Balanced (common default): temperature 0.7, top-p 0.95.

Common Mistakes

  • Cranking temperature and top-p high together → gibberish.
  • Using top-p 1.0 → no trimming at all (same as top-p off).
  • Expecting same output across runs — sampling is random by design; use temperature 0 for reproducible results.

Key Takeaways

  • Top-k trims the candidate list; top-p trims by cumulative probability.
  • top-p adapts to confidence automatically — usually the better dial.
  • Combine: temperature for creativity, top-p for sanity.
  • For reproducible output (tests, agents), temperature = 0.

Next up: Hallucinations — why models invent facts and how to stop it.

Interactive Lesson Code Snippet
# Top-p (nucleus) vs top-k sampling in action
tokens = ["Paris", "London", "Berlin", "Madrid", "Rome", "Tokyo"]
probs = [0.55, 0.18, 0.12, 0.08, 0.04, 0.03]

def top_p(probs, p):
    ordered = sorted(zip(tokens, probs), key=lambda x: -x[1])
    cumulative = 0.0
    selected = []
    for token, prob in ordered:
        cumulative += prob
        selected.append(token)
        if cumulative >= p:
            break
    return selected, cumulative

def top_k(probs, k):
    ordered = sorted(zip(tokens, probs), key=lambda x: -x[1])
    return [t for t, _ in ordered[:k]]

for p_value in [0.9, 0.95]:
    selected, cumulative = top_p(probs, p_value)
    print(f"Top-p = {p_value}: keep {selected} (cumulative {cumulative:.2f})")

print(f"\nTop-k = 3: keep {top_k(probs, 3)}")
print(f"Top-k = 5: keep {top_k(probs, 5)}")
Language: python

Code Visualization Tips

  • Draw a probability bar chart and shade the top-p area — the dynamic cutoff is the whole idea.
  • Draw the same chart with a top-k line — a fixed cutoff that ignores the shape of the tail.
  • Label a diagram 'top-k → top-p → temperature → sample' to remember the pipeline order.

Professional Tips & Tricks

  • Prefer top-p over top-k in most APIs — it adapts to how confident the model is.
  • If output contains garbage tokens, lower top-k to 40–60 rather than raising temperature.
  • Document your sampling settings per task — teams waste hours rediscovering the right combo.

Python Code Judge & Practice Arena

LeetCode Style

Run real Python 3.12 WebAssembly code directly in your browser against automated test suites.

Solved:0 / 1
0 / 20 XP
Challenges:
Problem 1 of 1

Top-P Cutoff Calculation

Medium+20 XP
Given probabilities [0.50, 0.25, 0.15, 0.10], which tokens are kept at top_p = 0.8? At top_p = 0.9?
main.pyPython 3.12 (WASM)
1
2
3
4
5
6
7
8
9
10
11
12
Press Run Code to test or Submit to verify test cases

Up next · Continue learning

Why LLMs Hallucinate

Hallucinations are a feature of how LLMs work, not a bug you can switch off. Understand the root causes.

9 mins read45 mins
Start next lesson
Temperature Explained Why LLMs Hallucinate
Made withbyAmol Shukla·amolshukla.online