Lesson 22: Iterators, Generators & itertools

Iteration Without Memory Blowups

A generator produces values one at a time using yield. Unlike a list, it does not store everything in memory at once — perfect for huge or even infinite sequences.

This is the lesson where your Python goes from "works on small data" to "handles real-world data". Generators are the reason Python can process files larger than RAM, stream infinite sequences, and feed machine-learning pipelines without memory crashes.

What You'll Learn in This Lesson

• Write generators with yield
• Understand lazy evaluation
• Use generator expressions
• Apply itertools building blocks

yield — Pause and Resume

When a function contains yield, calling it returns a generator object. Each next() runs the code until the next yield, then pauses — saving all local state:

def countdown(n):
    while n >= 0:
        yield n
        n -= 1

for num in countdown(3):     # 3 2 1 0
    print(num)

Mental model: a generator is a book that reveals one page at a time. You read a page, close the book, and reopen exactly where you left off. The bookmark (local state) is saved.

Watch the difference between return and yield:

Why Generators Matter

Processing a 10 GB log file: a list loads all 10 GB into memory; a generator streams line by line using kilobytes.

big_list = [x * 2 for x in range(10_000_000)]   # builds 10M items — ~89 MB
big_gen  = (x * 2 for x in range(10_000_000))   # builds nothing yet — 112 bytes

Data pipelines, streaming, and AI training loops all rely on this lazy pattern.

The one-shot rule: a generator is a conveyor belt, not a warehouse. Once you've taken every item, the belt is empty — iterate again and you get nothing. If you need two passes, convert to a list (and pay the memory cost) or rebuild the generator.

Generator Expressions

(x * 2 for x in range(10)) — a lazy comprehension. Pass it straight into sum(), max(), or a loop:

total = sum(x * x for x in range(1, 101))   # 338350 — no giant list

List vs generator expression — which to use?

itertools — The Toolbox

itertools ships with powerful iteration building blocks:

from itertools import islice, chain
print(list(islice((x for x in range(100) if x % 2 == 0), 5)))  # first 5 evens
print(list(chain([1, 2], [3, 4])))                              # [1, 2, 3, 4]

Peeking at an infinite sequence safely:

from itertools import count, islice
evens = count(0, 2)                          # 0, 2, 4, 6, ... forever
print(list(islice(evens, 5)))                # [0, 2, 4, 6, 8] — take 5, no crash

Mental model: islice is a "take n items and stop" guard — the only safe way to look at an infinite generator.

Real-World Generator Patterns

• Reading huge files: for line in open("big.log") — one line in memory at a time (Lesson 20).
• API pagination: a generator that fetches the next page only when asked.
• Infinite data: count(), cycle() for round-robin load balancing and game logic.
• Pipelines: chain generators — clean(parse(raw(f))) — each stage lazy, total memory O(1).

Common Mistakes to Avoid

• Mistake: Iterating a generator twice and getting nothing the second time — Fix: generators are one-shot; rebuild or convert to a list if you need two passes.
• Mistake: Mixing yield and return value in the same function — Fix: in a generator, return ends iteration (and return value is invalid); use yield for values.
• Mistake: Materializing huge lists when a generator would do — Fix: reach for (expr for ...) or itertools.
• Mistake: Forgetting the parentheses in a generator expression passed to a function — Fix: sum(x for x in ...) is fine without extra parens when it's the only argument.
• Mistake: Calling len() or indexing a generator — Fix: generators have no length and no indexes; convert to a list first if you need them.

Professional Tips & Tricks

• Use generators for file lines, API pagination, and any stream — never materialize huge lists.
• Prefer (expr for ...) over [expr for ...] when passing to sum/max/any.
• itertools.islice lets you peek at infinite generators safely.
• Chain lazy stages into pipelines: each stage yields, total memory stays O(1).
• Remember the one-shot rule: rebuild generators for a second pass.

Key Takeaways

• yield creates a lazy, stateful generator.
• Generators use O(1) memory and are one-shot.
• Generator expressions (expr for ...) feed into sum/max/any.
• itertools provides islice, chain, product, and more.
• islice makes infinite generators safe to peek at.

Next up: JSON & working with real-world data.