Lesson 5: Managing Long Contexts
Chunking, sliding windows, summarization, and retrieval — the techniques that let you work with text far larger than any window.
The Problem
Your company handbook is 200,000 tokens. Your model's window is 128,000 tokens. You cannot paste the whole thing — so what do you do?
Four proven strategies, from simplest to most powerful:
1. Summarization (Compression)
Ask the model to summarize the document first, then work from the summary.
- Pros: simple, cheap, no infrastructure.
- Cons: loses details; summaries drift over many rounds.
2. Chunking (Splitting)
Split the document into overlapping chunks (e.g. 500–1,000 tokens with 50–100 tokens of overlap) and process each chunk separately.
- Overlap matters: it preserves sentence and paragraph boundaries so meaning isn't cut in half.
- Use structural boundaries (paragraphs, sections, markdown headers) instead of blind character counts when possible.
3. Sliding Window (Streaming)
For conversation, keep only the last N messages and drop (or summarize) older ones. This is how chat apps keep long conversations fast and cheap.
4. Retrieval-Augmented Generation (RAG)
Keep the full document in a vector database, and for each question retrieve only the most relevant chunks into the context window. This is the professional standard — covered in depth in Lesson 13.
Choosing a Strategy
| Situation | Strategy |
|---|---|
| One-off analysis of a big document | Summarize or chunk + summarize |
| Long-running chat | Sliding window + summarization |
| Question-answering over a knowledge base | RAG (retrieval) |
| Massive codebase | RAG over code chunks with repo maps |
Chunking Recipe (that actually works)
- Split on semantic boundaries: markdown headings, paragraphs, function/class definitions in code.
- Use overlap (10–20%) so context isn't lost at seams.
- Aim for chunks of 300–800 tokens for retrieval; larger chunks for summarization.
- Test chunk size against retrieval quality — measure, don't guess.
Key Takeaways
- Summarize, chunk, slide, or retrieve — pick the tool for the job.
- Chunk with overlap and respect semantic boundaries.
- RAG is the professional answer to "my data is bigger than the window".
- Long contexts cost more and can reduce focus — don't use 128K when 8K will do.
Next up: Temperature and sampling — how the model chooses between safe and creative answers.
# Chunking a long document so it fits inside a context window
def chunk_text(text, chunk_size=500, overlap=50):
chunks = []
start = 0
while start < len(text):
end = min(start + chunk_size, len(text))
chunks.append(text[start:end])
if end == len(text): # reached the end - stop (avoid infinite loop)
break
start = end - overlap
return chunks
# Build a long "document" by repeating a sentence 100 times
document = " ".join(["Large language models predict the next token."] * 100)
chunks = chunk_text(document)
print(f"Document length: {len(document)} characters")
print(f"Chunk size: 500 chars | Overlap: 50 chars")
print(f"Chunks created: {len(chunks)}")
total_chars = sum(len(c) for c in chunks)
print(f"Average chunk size: {total_chars // len(chunks)} chars")
print(f"First chunk starts: \"{chunks[0][:25]}...\"")
print(f"Last chunk ends: \"...{chunks[-1][-25:]}\"")Lesson Code (Python)
# Chunking a long document so it fits inside a context window
def chunk_text(text, chunk_size=500, overlap=50):
chunks = []
start = 0
while start < len(text):
end = min(start + chunk_size, len(text))
chunks.append(text[start:end])
if end == len(text): # reached the end - stop (avoid infinite loop)
break
start = end - overlap
return chunks
# Build a long "document" by repeating a sentence 100 times
document = " ".join(["Large language models predict the next token."] * 100)
chunks = chunk_text(document)
print(f"Document length: {len(document)} characters")
print(f"Chunk size: 500 chars | Overlap: 50 chars")
print(f"Chunks created: {len(chunks)}")
total_chars = sum(len(c) for c in chunks)
print(f"Average chunk size: {total_chars // len(chunks)} chars")
print(f"First chunk starts: \"{chunks[0][:25]}...\"")
print(f"Last chunk ends: \"...{chunks[-1][-25:]}\"")Console Output
Document length: 4599 characters
Chunk size: 500 chars | Overlap: 50 chars
Chunks created: 11
Average chunk size: 463 chars
First chunk starts: "Large language models pre..."
Last chunk ends: "...s predict the next token."Code Visualization Tips
- Draw the document as a long strip, then draw the chunks with their overlapping seams.
- Animate the sliding window over a chat transcript to show which messages are in view.
- Diagram the RAG flow: Document → Chunks → Vector DB → Retrieve → Prompt → Answer.
Professional Tips & Tricks
- Chunk on paragraph/heading boundaries, not fixed character counts — your retrieval quality will jump.
- Keep 10–20% overlap; zero overlap silently cuts meaning at every seam.
- For code, chunk per function/class; for docs, chunk per section — structure is a free semantic signal.
Python Code Judge & Practice Arena
LeetCode StyleRun real Python 3.12 WebAssembly code directly in your browser against automated test suites.
Overlap Reasoning
Up next · Continue learning
Temperature Explained
What temperature actually does to the probability distribution, and when to use low, medium, or high values.