Lesson 2: The Transformer & Self-Attention
Peek inside the model: embeddings, the transformer stack, and the self-attention mechanism that decides which words matter.
The Transformer Architecture
Almost every modern LLM is a transformer, introduced in the 2017 paper "Attention Is All You Need". The transformer's superpower is self-attention: it lets every token in the input "look at" every other token and decide how much each one matters.
Why Attention Was a Breakthrough
Earlier models (like RNNs) processed text left-to-right and struggled with long-range connections. In the sentence:
"The bank guaranteed a full refund…"
…the word bank needs to know about guaranteed and refund — words far away. Self-attention gives every token a direct connection to every other token, in a single pass.
Inside a Transformer Layer
| Component | Job |
|---|---|
| Embeddings | Convert tokens into numeric vectors |
| Positional encoding | Add information about each token's position |
| Self-attention | Compute weighted relationships between all tokens |
| Multi-head attention | Run several attention patterns in parallel (different "lenses") |
| Feed-forward network | Process each token's representation further |
| Layer normalization | Keep numbers stable so training works |
The whole stack is repeated dozens of times (GPT-4-class models have 100+ layers and billions of parameters).
Attention in One Formula
Attention scores are computed by comparing a query (what am I looking for?) with keys (what do I contain?), then using values (what should I contribute?):
Attention(Q, K, V) = softmax\left(\frac{QK^T}{\sqrt{d_k}}\right)V
Don't worry about the math — the intuition is: "for each word, how relevant is every other word?"
Decoder-Only vs. Encoder-Decoder
| Family | Example | Best For |
|---|---|---|
| Decoder-only | GPT, Llama, Claude | Text generation, chat, agents |
| Encoder-decoder | T5, FLAN | Translation, summarization |
| Encoder-only | BERT | Embeddings, classification |
Most AI tools you use today are decoder-only autoregressive models.
Key Takeaways
- Transformers use self-attention so every token can relate to every other token.
- Multi-head attention runs many "lenses" in parallel, capturing different relationships.
- Decoder-only models (GPT-style) dominate modern AI tools.
- Attention is what makes long-range understanding (and long context windows) possible.
Next up: How these models get trained — pre-training, fine-tuning, and RLHF.
# Tiny self-attention: how much should each word "look at" the others?
words = ["the", "bank", "guaranteed", "a", "refund"]
# Made-up attention weights for the query word "bank"
# (in a real model these come from Q·K similarity, learned during training)
attention = {"the": 0.05, "bank": 0.08, "guaranteed": 0.52, "a": 0.07, "refund": 0.28}
print(f"Context: {' '.join(words)}")
print("How strongly does 'bank' attend to each word?\n")
for word, weight in attention.items():
bar = "#" * int(weight * 40)
print(f" {word:12s} {weight:.2f} {bar}")
total = sum(attention.values())
print(f"\nWeights sum to: {total:.2f}")
# The model would now blend the *values* of all words using these weights
print("-> 'bank' is mostly influenced by 'guaranteed' and 'refund'.")Lesson Code (Python)
# Tiny self-attention: how much should each word "look at" the others?
words = ["the", "bank", "guaranteed", "a", "refund"]
# Made-up attention weights for the query word "bank"
# (in a real model these come from Q·K similarity, learned during training)
attention = {"the": 0.05, "bank": 0.08, "guaranteed": 0.52, "a": 0.07, "refund": 0.28}
print(f"Context: {' '.join(words)}")
print("How strongly does 'bank' attend to each word?\n")
for word, weight in attention.items():
bar = "#" * int(weight * 40)
print(f" {word:12s} {weight:.2f} {bar}")
total = sum(attention.values())
print(f"\nWeights sum to: {total:.2f}")
# The model would now blend the *values* of all words using these weights
print("-> 'bank' is mostly influenced by 'guaranteed' and 'refund'.")Console Output
Context: the bank guaranteed a refund
How strongly does 'bank' attend to each word?
the 0.05 ##
bank 0.08 ###
guaranteed 0.52 ####################
a 0.07 ##
refund 0.28 ###########
Weights sum to: 1.00
-> 'bank' is mostly influenced by 'guaranteed' and 'refund'.Code Visualization Tips
- Draw an attention heatmap: words on both axes, cell darkness = attention weight.
- Use the bar chart output to explain how 'bank' disambiguates between financial and river meaning.
- Sketch a transformer layer as a box with arrows: Embeddings → Attention → Feed-forward → out.
Professional Tips & Tricks
- When a model misreads ambiguous words, restate the context — attention shifts with your words.
- Long, clear sentences help attention spread evenly; confusing prompts create noisy attention.
- You don't need to compute attention yourself — but knowing it exists explains why context order matters.
Python Code Judge & Practice Arena
LeetCode StyleRun real Python 3.12 WebAssembly code directly in your browser against automated test suites.
Attention & Ambiguity
Up next · Continue learning
The Training Pipeline — Pre-training, Fine-tuning & RLHF
How a raw neural network becomes ChatGPT: pre-training on trillions of tokens, supervised fine-tuning, and reinforcement learning from human feedback.