Python Generators

A generator is a special kind of Python function that produces a sequence of values lazily, one at a time, instead of computing and returning them all at once. Generators let you write memory-efficient, readable code for iterating over large or even infinite sequences of data, because each value is produced only when it is requested. Under the hood, a generator is really just an iterator that Python builds automatically from a function that contains the yield keyword. Once you understand how yield pauses and resumes a function’s execution, generators become one of the most useful tools in the Python language.

Overview: How Generators Work

A normal function runs from start to finish every time you call it, computes a result, and returns it with a single return statement — after that, the function’s local state is gone. A generator function behaves completely differently. If a function’s body contains the keyword yield anywhere, Python treats the entire function specially: calling it does not run any of the code inside. Instead, it immediately returns a generator object, which is an iterator that remembers exactly where execution should resume.

Each time you ask the generator for the next value (via next(), a for loop, or unpacking), Python resumes the function’s body from wherever it last left off, runs until it hits a yield expression, and pauses again — handing back the yielded value. All local variables, the instruction pointer, and the call stack for that function are preserved in between calls. This is what makes generators special: they are functions with memory, capable of being paused and resumed on demand.

When the function body eventually finishes (falls off the end or hits a bare return), the generator raises a StopIteration exception internally, which is what tells a for loop to stop. Because generators are iterators, they implement the iterator protocol: a __iter__() method that returns themselves, and a __next__() method that advances execution. This also means generators are single-use and exhaustible — once you’ve consumed all the values, the generator is empty and cannot be restarted; you must create a new one.

The biggest practical benefit is lazy evaluation. A list comprehension like [x * x for x in range(1_000_000)] builds the entire million-element list in memory immediately. The equivalent generator expression, (x * x for x in range(1_000_000)), produces each square only when requested, using a small constant amount of memory no matter how many items are involved. This makes generators ideal for streaming data, reading huge files line by line, or representing infinite sequences.

Syntax

A generator function looks like an ordinary function definition, except its body contains one or more yield statements:

def generator_name(parameters):
    # ... setup code ...
    yield value
    # ... more code ...
    yield another_value

There is also a shorthand called a generator expression, syntactically similar to a list comprehension but written with parentheses instead of square brackets: (expression for item in iterable if condition).

Concept	Purpose
yield value	Pauses the function and sends value out to the caller; execution resumes right after this line on the next request.
next(gen)	Resumes the generator until the next yield, or raises StopIteration if the generator is finished.
gen.send(value)	Resumes the generator, making the paused yield expression evaluate to value; also advances to the next yield.
gen.throw(exc)	Raises an exception inside the generator at the paused yield point.
gen.close()	Stops the generator by raising GeneratorExit inside it.
yield from sub_gen	Delegates iteration to another iterable or generator, yielding all of its values in turn.

Examples

Example 1: Manual iteration with next()

def simple_gen():
    yield "first"
    yield "second"
    yield "third"

gen = simple_gen()
print(next(gen))
print(next(gen))
print(next(gen))

Output:

first
second
third

Calling simple_gen() does not run any code — it just creates a generator object. Each call to next() resumes execution until the next yield statement, returning that value and pausing again. If you called next(gen) a fourth time, it would raise StopIteration because the function has no more code to run.

Example 2: A Fibonacci generator

def fibonacci(limit):
    a, b = 0, 1
    while a < limit:
        yield a
        a, b = b, a + b

fib_numbers = list(fibonacci(50))
print(fib_numbers)

Output:

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34]

This generator could just as easily produce Fibonacci numbers forever by removing the limit check — something a regular function returning a list could never do, since it would try to build an infinitely long list and never return. Here, wrapping the call in list() forces the generator to run to completion and collects every yielded value.

Example 3: A lazy data-processing pipeline

def read_records(records):
    for record in records:
        yield record.strip()

def parse_ints(lines):
    for line in lines:
        if line.isdigit():
            yield int(line)

def squares(numbers):
    for number in numbers:
        yield number ** 2

raw_data = [" 3 ", "abc", " 5 ", "10", " "]
pipeline = squares(parse_ints(read_records(raw_data)))
print(list(pipeline))

Output:

[9, 25, 100]

This is a realistic use of generators: chaining several small, single-purpose generator functions into a pipeline. No intermediate list is ever built — each raw record flows through read_records, then parse_ints, then squares, one item at a time. This pattern scales effortlessly to huge inputs, such as processing gigabytes of log lines, because at any moment only one record is being held in memory across the whole pipeline.

How It Works Step by Step / Under the Hood

Generators also support two-way communication through send(), which lets the caller push a value into the paused yield expression:

def running_total():
    total = 0
    while True:
        value = yield total
        total += value

accumulator = running_total()
next(accumulator)
print(accumulator.send(10))
print(accumulator.send(5))
print(accumulator.send(20))

Output:

10
15
35

The call to next(accumulator) is required first to “prime” the generator — it runs the function up to its first yield total (yielding 0, which we discard). From then on, each send(value) does two things at once: it supplies value as the result of the paused yield expression (assigned to the value variable), and it resumes execution until the next yield, whose result becomes send()‘s return value. This is how coroditional-style generators maintain running state across calls.

Generators can also delegate work to other generators with yield from, which flattens nested iteration automatically:

def inner_generator():
    yield 1
    yield 2
    yield 3

def outer_generator():
    yield "start"
    yield from inner_generator()
    yield "end"

for value in outer_generator():
    print(value)

Output:

start
1
2
3
end

yield from inner_generator() is roughly equivalent to writing for item in inner_generator(): yield item, but it also properly forwards send(), throw(), and the sub-generator’s return value, which the manual loop would not do.

Common Mistakes

Mistake 1: Reusing an exhausted generator

numbers = (x * x for x in range(5))
total = sum(numbers)
print(f"Total: {total}")

try:
    maximum = max(numbers)
except ValueError as error:
    print(f"Error: {error}")

Output:

Total: 30
Error: max() arg is an empty sequence

sum(numbers) fully consumes the generator expression. By the time max(numbers) runs, there is nothing left to iterate, so it raises ValueError. Generators are not reusable containers like lists. Fix this by materializing the values into a list once if you need to use them more than once, or by creating a fresh generator for each pass: numbers = list(x * x for x in range(5)), then call sum(numbers) and max(numbers) on that list.

Mistake 2: Assuming a generator function runs immediately

def process_data(items):
    print("Starting processing")
    for item in items:
        yield item * 2

gen = process_data([1, 2, 3])
print("Generator object created")
for value in gen:
    print(value)

Output:

Generator object created
Starting processing
2
4
6

New learners often expect "Starting processing" to print the moment process_data(...) is called. It doesn’t — calling a generator function only creates the generator object; none of the function body runs until iteration actually begins. This is a direct consequence of lazy evaluation, but it can be surprising when a generator wraps code with side effects like logging or file opening.

A related trap: generators are not sequences. You cannot index them with gen[0] or call len(gen) — both raise a TypeError, because a generator only knows how to produce the next value, not any arbitrary one. If you need indexing, length, or multiple passes, convert it to a list with list(gen) first.

Best Practices

• Use a generator (function or expression) whenever you only need to iterate over data once and don’t need the whole sequence in memory at the same time.
• Prefer a generator expression like (x for x in data) over a list comprehension when passing data straight into functions such as sum(), any(), all(), or max() — you save the memory of building an intermediate list.
• Give generator functions descriptive verb-based names (e.g. read_lines, filter_valid) since they represent an ongoing process, not a single value.
• Chain small, focused generator functions into pipelines instead of writing one large function that does everything — it keeps each stage testable and composable.
• Remember to “prime” a generator with next() before the first send() call if the generator needs to receive values via yield expressions.
• Use yield from instead of a manual for ... yield loop when delegating to a sub-generator — it’s shorter and correctly forwards send()/throw()/return values.
• Convert a generator to a list explicitly (list(gen)) only when you truly need random access, its length, or to iterate over it more than once.
• Document whether a function returning a generator can be iterated more than once — by default, assume it cannot.

Practice Exercises

• Exercise 1: Write a generator function evens_up_to(n) that yields every even number from 0 up to (and including) n. Test it by printing list(evens_up_to(10)).
• Exercise 2: Write a generator function countdown(n) that yields n, n-1, n-2, ..., 1 and then the string "Liftoff!" as its final yielded value. Iterate over it with a for loop and print each value.
• Exercise 3: Write two generator functions, positive_numbers(numbers) that yields only values greater than zero, and double_values(numbers) that yields each input value multiplied by two. Chain them together as double_values(positive_numbers(data)) for data = [-3, 5, -1, 8, 0, 2] and print the result as a list. (Expected output: [10, 16, 4].)

Summary

• A generator function is any function containing yield; calling it returns a generator object without running the body.
• Each next() call resumes execution up to the next yield, preserving local state in between calls, until the function ends and StopIteration is raised.
• Generator expressions, written as (expr for item in iterable), are a lazy, memory-efficient alternative to list comprehensions.
• Generators are single-use iterators — once exhausted, they must be recreated, and they don’t support indexing or len().
• send() allows two-way communication with a paused generator; yield from cleanly delegates to a sub-generator.
• Chaining generators into pipelines processes data lazily, one item at a time, which scales well to large or streaming datasets.