45 mins lesson duration•9 mins read

Lesson 20: Self-Consistency — Sample & Vote

Run the same prompt several times and take the majority answer to lift accuracy on hard reasoning tasks, with examples.

The Problem with One Answer

Chain-of-thought helps — but the model can still pick a wrong reasoning path. Self-consistency runs the same reasoning prompt multiple times and takes the majority answer. Wrong paths are diverse; the right path repeats.

The Recipe

Ask the question with chain-of-thought.
Generate 3–5 independent answers (use a higher temperature like 0.5–0.7 so paths vary).
Cluster the answers and pick the most common one (majority vote).

Worked Example

Question: A store sells shirts for $25. A 20% discount applies when
you buy 3 or more. How much do 5 shirts cost? Think step by step.

Run it 5 times (temperature 0.6):

Run	Reasoning path	Answer
1	5 shirts → discount applies → 20% of 25 = 5 → 20 × 5	100
2	Discount applies → 25 × 5 = 125 → 125 − 25	100
3	Forgot discount → 25 × 5	125
4	20% off 25 = 5 → 20 each → 5 × 20	100
5	Discount only on 3 → 3 × 20 + 2 × 25	110

Majority = 100 — the correct answer wins even though runs 3 and 5 went wrong.

When to Use It

Situation	Use Self-Consistency?
Math, logic, multi-step reasoning	✅ Yes — biggest gains here
Factual recall (names, dates)	⚠️ Rarely helps — all runs share the same wrong memory
Creative writing	❌ No — you want variety, not a vote
Production APIs with budget	⚠️ 3–5x cost and latency; use only for critical answers

The Cost Trade-Off

Runs	Accuracy gain	Cost
1	baseline	1x
3	good	3x
5	diminishing returns	5x

Rule of thumb: start with 3 runs for high-stakes reasoning, and only escalate if the vote is split.

Common Mistakes

Voting on creative outputs — there is no "correct" creative answer.
Using temperature 0 — all runs give the same answer, so the vote is pointless.
Taking the "most common" of free-form essays — self-consistency works best with short, extractable answers (a number, a label, a one-line conclusion).

Key Takeaways

Self-consistency = multiple CoT runs + majority vote.
It fixes reasoning errors, not memory errors.
Works best with short, comparable answers; costs 3–5x.
Use 3 runs first; escalate only when the vote is split.

Next up: Tree of thought and persona prompting — exploring branches and assigning expertise.

Interactive Lesson Code Snippet

# Self-consistency: sample several answers, take the majority
import random
from collections import Counter

answers = []

def sample_answer(seed):
    rng = random.Random(seed)
    # Simulates 5 CoT runs: correct answer 70% of the time
    return "21" if rng.random() < 0.7 else "20"

for i in range(5):
    ans = sample_answer(i)
    answers.append(ans)
    print(f"Sample {i+1}: {ans}")

vote = Counter(answers).most_common(1)[0]
print(f"\nMajority vote: {vote[0]} (appeared {vote[1]} times)")

Language: python

Lesson Code (Python)

# Self-consistency: sample several answers, take the majority
import random
from collections import Counter

answers = []

def sample_answer(seed):
    rng = random.Random(seed)
    # Simulates 5 CoT runs: correct answer 70% of the time
    return "21" if rng.random() < 0.7 else "20"

for i in range(5):
    ans = sample_answer(i)
    answers.append(ans)
    print(f"Sample {i+1}: {ans}")

vote = Counter(answers).most_common(1)[0]
print(f"\nMajority vote: {vote[0]} (appeared {vote[1]} times)")

Console Output

Sample 1: 20
Sample 2: 21
Sample 3: 20
Sample 4: 21
Sample 5: 21

Majority vote: 21 (appeared 3 times)

Code Visualization Tips

Draw 5 reasoning paths as arrows from the question; the majority arrow wins the vote.
Make a tally chart of answers across runs — the 'cluster' around the correct one is the signal.
Sketch the cost ladder: 1 run = 1x, 3 runs = 3x, 5 runs = 5x with accuracy plateauing.

Professional Tips & Tricks

Extract the final answer as a single line ('Answer: ...') so votes are easy to compare.
Use temperature 0.5–0.7 for the samples — too low gives clones, too high gives noise.
Skip self-consistency for factual questions; ground those with RAG instead.

Python Code Judge & Practice Arena

LeetCode Style

Run real Python 3.12 WebAssembly code directly in your browser against automated test suites.

Solved:0 / 1

0 / 20 XP

Challenges:

Problem 1 of 1

Vote on Reasoning Paths

Medium+20 XP

Below are 4 CoT answers to 'A pizza costs $12. A 10% tip is added. What is the total?' — answer with the majority: (a) 13.2, (b) 13.2, (c) 12.1, (d) 13.2.

main.pyPython 3.12 (WASM)

# Vote on Reasoning Paths
# Write your solution below

def solve():
    pass

Press Run Code to test or Submit to verify test cases

Test Your Knowledge

Instant feedback

Quick Check: Self-Consistency

1 / 3

What does self-consistency do?

Up next · Continue learning

Tree of Thought & Persona Prompting

Explore several reasoning branches before deciding, and assign the model a persona to unlock specialized perspectives — with examples.

10 mins read50 mins

Start next lesson

Previous: Prompting for Agentic Workflows Next: Tree of Thought & Persona Prompting

Made withbyAmol Shukla·amolshukla.online