LLM Sampling Techniques: Minimum Probability and Temperature

Minimum Probability Sampling

Definition

Minimum probability sampling is a technique used in language model APIs to balance between diversity and coherence in the model’s output.

How it works

• Sets a dynamic threshold for token selection based on the probability of the most likely token.
• The threshold is a fraction (determined by the min_p value) of the top token’s probability.

Example explanation

Let’s say min_p = 0.1, and we’re generating the next token:

Scenario A:

• Most likely token probability: 95%
• Threshold: 95% * 0.1 = 9.5%
• Only tokens with probabilities ≥ 9.5% are considered

Scenario B:

• Most likely token probability: 10%
• Threshold: 10% * 0.1 = 1%
• Tokens with probabilities ≥ 1% are considered

Adaptive nature

• When the model is very confident (high top probability), the threshold is higher, limiting options to maintain coherence.
• When the model is less certain (lower top probability), the threshold lowers, allowing more diverse options.

Benefits

• Preserves diversity for open-ended choices
• Maintains coherence for deterministic choices (e.g., programming syntax)
• Allows higher temperatures without losing coherence

Temperature in LLM Sampling

Definition

Temperature controls the randomness in token selection during text generation.

Effects of Higher Temperature

1. Increased diversity in outputs
2. Exploration of less likely options
3. Reduced repetitiveness
4. Better performance on open-ended tasks
5. Potential mitigation of model biases
6. Improved resilience to prompt engineering

Challenges

• Maintaining coherence and relevance at higher temperatures

Optimal Use

• Lower temperatures: Tasks requiring high accuracy or factual correctness
• Higher temperatures: Creative or exploratory tasks

Synergy: min_p and Temperature

Combining min_p sampling with higher temperatures allows for:

• Increased creativity and diversity in outputs
• Maintained coherence by filtering out extremely improbable tokens

Key Takeaways

1. min_p sampling adapts token selection threshold based on the model’s confidence.
2. Higher temperatures increase output diversity but risk coherence.
3. Combining min_p with higher temperatures balances creativity and coherence.
4. The optimal sampling strategy depends on the specific task and desired outcome.