60 mins lesson duration•11 mins read

Lesson 3: The Training Pipeline — Pre-training, Fine-tuning & RLHF

How a raw neural network becomes ChatGPT: pre-training on trillions of tokens, supervised fine-tuning, and reinforcement learning from human feedback.

How LLMs Get Their Skills

An LLM is not programmed — it is trained. The journey from random weights to ChatGPT happens in a pipeline of stages, each with a different goal.

Stage 1: Pre-training (The Autocomplete Phase)

The model is fed trillions of tokens scraped from the internet, books, code, and papers. Its only task: predict the next token. Through backpropagation, it slowly adjusts billions of weights to reduce prediction error.

Cost: astronomical — thousands of GPUs for months (millions of dollars).
Result: a base model that is great at language but not yet "helpful".
Knowledge cutoff: everything the model "knows" was in its training data. If the data ends in early 2025, the model doesn't know 2026 events.

Stage 2: Supervised Fine-Tuning (SFT) (The Follow-Instructions Phase)

The base model is fine-tuned on curated instruction–response pairs written by humans: "Summarize this email" → a great summary. This teaches the model to follow instructions and answer in a helpful format.

Stage 3: RLHF (The Alignment Phase)

Reinforcement Learning from Human Feedback makes the model prefer answers humans like:

The model generates multiple answers to the same prompt.
Human raters rank them: "this one is helpful and safe."
A reward model learns to predict those rankings.
The LLM is fine-tuned (via PPO/DPO) to maximize the reward.

This is why ChatGPT answers feel polite, structured, and safe compared to raw base models.

The Full Pipeline

Stage	Input	Goal	Who Does It
Pre-training	Trillions of raw tokens	Predict next token	Big labs (OpenAI, Anthropic, Meta…)
SFT	Instruction–answer pairs	Follow instructions	Labs + open-source community
RLHF	Human preference rankings	Be helpful & safe	Labs
Fine-tuning (yours)	Your domain data	Specialize for your task	You

What This Means for You as a User

The model's "knowledge" is frozen at training time — it cannot browse or learn unless you give it tools (RAG, web search, agents).
Fine-tuning (teaching the model your style/domain) is usually the last lever you pull — first try better prompts, then RAG, then fine-tuning.
The training pipeline is why models are confident, fluent, and sometimes wrong: fluency comes from pre-training, but truth was never a training objective.

Key Takeaways

Pre-training teaches language; SFT teaches helpfulness; RLHF teaches alignment.
Knowledge is frozen at the training data cutoff — use tools for current facts.
Better prompts → RAG → fine-tuning, in that order, before retraining anything.

Next up: The context window — how much text the model can actually see at once.

Interactive Lesson Code Snippet

# Simulating pre-training: cross-entropy loss drops as the model learns
epochs = list(range(1, 11))
loss = [4.80, 4.31, 3.94, 3.65, 3.42, 3.24, 3.09, 2.97, 2.87, 2.79]

print("Pre-training loss over 10 epochs (simulated):\n")
for epoch, value in zip(epochs, loss):
    bar = "#" * int(value * 10)
    print(f"  Epoch {epoch:2d}  loss={value:.2f}  {bar}")

print("\nLower loss = better next-token prediction.")
print("Real models train for far longer on far more data.")

# The three stages of the pipeline, in order
pipeline = ["1. Pre-training (predict next token)",
            "2. Supervised fine-tuning (follow instructions)",
            "3. RLHF (prefer helpful, safe answers)"]
print("\nTraining pipeline:")
for stage in pipeline:
    print("  " + stage)

Language: python

Lesson Code (Python)

# Simulating pre-training: cross-entropy loss drops as the model learns
epochs = list(range(1, 11))
loss = [4.80, 4.31, 3.94, 3.65, 3.42, 3.24, 3.09, 2.97, 2.87, 2.79]

print("Pre-training loss over 10 epochs (simulated):\n")
for epoch, value in zip(epochs, loss):
    bar = "#" * int(value * 10)
    print(f"  Epoch {epoch:2d}  loss={value:.2f}  {bar}")

print("\nLower loss = better next-token prediction.")
print("Real models train for far longer on far more data.")

# The three stages of the pipeline, in order
pipeline = ["1. Pre-training (predict next token)",
            "2. Supervised fine-tuning (follow instructions)",
            "3. RLHF (prefer helpful, safe answers)"]
print("\nTraining pipeline:")
for stage in pipeline:
    print("  " + stage)

Console Output

Pre-training loss over 10 epochs (simulated):

  Epoch  1  loss=4.80  ################################################
  Epoch  2  loss=4.31  ###########################################
  Epoch  3  loss=3.94  #######################################
  Epoch  4  loss=3.65  ####################################
  Epoch  5  loss=3.42  ##################################
  Epoch  6  loss=3.24  ################################
  Epoch  7  loss=3.09  ##############################
  Epoch  8  loss=2.97  #############################
  Epoch  9  loss=2.87  ############################
  Epoch 10  loss=2.79  ###########################

Lower loss = better next-token prediction.
Real models train for far longer on far more data.

Training pipeline:
  1. Pre-training (predict next token)
  2. Supervised fine-tuning (follow instructions)
  3. RLHF (prefer helpful, safe answers)

Code Visualization Tips

Chart the loss curve on graph paper — the flattening tail shows diminishing returns.
Draw the pipeline as 3 boxes with arrows, labeling who does each stage and what the input is.
Annotate a chat response with 'this fluency came from pre-training' and 'this helpfulness came from RLHF'.

Professional Tips & Tricks

Check a model's knowledge cutoff before asking about recent events — or give it the facts yourself.
If a model is confident but wrong, remember: 'truth' was never its training objective.
Use fine-tuned open models for domain work only after prompts and RAG have been exhausted.

Python Code Judge & Practice Arena

LeetCode Style

Run real Python 3.12 WebAssembly code directly in your browser against automated test suites.

Solved:0 / 1

0 / 10 XP

Challenges:

Problem 1 of 1

Match the Training Stage

Easy+10 XP

For each behavior, name the stage that creates it: (a) the model writes fluent English, (b) the model answers politely instead of rambling, (c) the model refuses harmful requests.

main.pyPython 3.12 (WASM)

# Match the Training Stage
# Write your solution below

def solve():
    pass

Press Run Code to test or Submit to verify test cases

Up next · Continue learning

What Is a Context Window?

The context window is everything the model can 'see' at once — input plus output. Understand limits, truncation, and memory.

9 mins read45 mins

Start next lesson

Previous: The Transformer & Self-Attention Next: What Is a Context Window?

Made withbyAmol Shukla·amolshukla.online