Foundations

Actuality

This article is still relevant because generators are one of those Python features that are easy to admire in theory and easy to misuse in practice. They are often introduced as a universal optimization, but the real question is narrower: when does laziness actually improve the code?

A generator is useful when it reduces memory pressure or expresses a streaming workflow more clearly. It is not automatically better than a list.

Thesis

Generators are most useful when values are produced progressively, consumed once, or potentially unbounded. They are much less useful when the data is already in memory or when the calling code benefits from an ordinary concrete collection.

In other words, a generator is not a sign of sophistication. It is a tool for specific shapes of work: streaming, filtering, pipelining, and stateful iteration.

Core concepts

Three distinctions matter here.

Existing data versus produced data. If the full collection already exists in memory, wrapping it in a generator often adds ceremony without real benefit.

One-pass consumption versus repeated access. A generator is ideal when data is consumed once in sequence. It is a poor fit when callers need indexing, reuse, or multiple passes.

Finite sequences versus open-ended streams. Generators are especially natural when the sequence is large, expensive to build, or conceptually unbounded.

Limits

Generators are not automatically more readable.

  • They can hide control flow when overused.
  • They are inconvenient when you need random access or repeated iteration.
  • They can make debugging harder because values do not exist all at once.
  • They do not help if the expensive part of the program is somewhere else.

The right question is not "can this be written as a generator?" but "does laziness improve this specific workflow?"

Implementation and examples

Case 1: iterating an existing sequence

If a list is already in memory, turning it into a generator expression just to loop over it does not buy much.

names = ["John", "Mary", "Mark"]

for name in names:
    print(name)

This is usually better than adding an unnecessary wrapper.

for name in (n for n in names):
    print(name)

The generator expression is not wrong, but it does not solve a real problem here. The data already exists.

Case 2: filtering while preserving order

Generators become more useful when you want to process a sequence progressively instead of building an intermediate collection.

Suppose you want to remove duplicates from a list while preserving the original order.

from collections.abc import Iterator


def deduplicated(data: list[str]) -> Iterator[str]:
    seen: set[str] = set()
    for element in data:
        if element not in seen:
            seen.add(element)
            yield element


many_names = ["John", "Mary", "John", "Mark"]

for name in deduplicated(many_names):
    print(name)

This is a good generator use case because values are produced one by one and can be consumed immediately.

The generator helps when the consumer wants a stream of cleaned values, not necessarily a finished list up front.

If the caller really needs a concrete result, it can still materialize one explicitly.

unique_names = list(deduplicated(many_names))

Case 3: reading files line by line

This is one of the clearest generator use cases. Files are naturally sequential, and reading them lazily keeps memory usage predictable.

A list-returning version loads everything at once:

def read_file(filename: str) -> list[str]:
    with open(filename, encoding="utf-8") as file:
        return [line.rstrip("\n") for line in file]

A generator version yields lines progressively:

from collections.abc import Iterator


def read_file(filename: str) -> Iterator[str]:
    with open(filename, encoding="utf-8") as file:
        for line in file:
            yield line.rstrip("\n")



for line in read_file("test.txt"):
    print(line)

This version scales much better when the file is large and fits the actual access pattern: read, process, move on.

Case 4: generating values on demand

Generators are also a natural fit when values do not exist yet and should be created only when requested.

For example, imagine an endless stream of random unique identifiers.

from collections.abc import Iterator
from random import choices
from string import ascii_letters


def unique_tokens(n: int = 10) -> Iterator[str]:
    seen: set[str] = set()
    while True:
        value = "".join(choices(ascii_letters, k=n))
        if value not in seen:
            seen.add(value)
            yield value


tokens = unique_tokens(10)
first = next(tokens)
second = next(tokens)

This is a strong generator case because the sequence is conceptually open-ended. There is no reason to precompute values you may never use.

Case 5: generating URLs progressively

Generators are useful when producing a large sequence of derived values that will be consumed one at a time.

from collections.abc import Iterator

BASE_URL = "https://my-blog.com/posts/"
N = 10_000


def get_urls(n: int) -> Iterator[str]:
    for i in range(1, n + 1):
        yield f"{BASE_URL}{i}"



import requests

for url in get_urls(N):
    response = requests.get(url)

The advantage here is not only memory. The generator also matches the shape of the task: produce one URL, use it, discard it, continue.

Case 6: generators as coroutines

There is also a more advanced use: generators can receive values through .send(...) and behave like simple coroutines.

Historically this mattered more before async and await, but it is still a useful idea for understanding Python's iteration model.

from collections.abc import Generator


def averager() -> Generator[float | None, float, None]:
    total = 0.0
    count = 0
    average: float | None = None

    while True:
        value = yield average
        total += value
        count += 1
        average = total / count

Usage:

coro = averager()
next(coro)  # prime the generator
print(coro.send(20))
print(coro.send(30))

This is interesting, but for most application code today, ordinary iterators or async code are more common practical tools.

Summary

Generators are best when data should be produced lazily, consumed once, or treated as a stream. They are much less compelling when the collection already exists or when callers need a regular container.

The important habit is to match the tool to the shape of the problem. Use generators when laziness clarifies the workflow. Use lists when concrete data is what the code actually wants.

Recommendations

  • do not wrap an existing in-memory list in a generator unless it solves a real problem
  • use generators for streaming workflows such as file reading and progressive URL generation
  • prefer generators when the sequence is large, expensive, or potentially infinite
  • return a list when callers need reuse, indexing, or multiple passes
  • treat generator syntax as a design choice, not as an automatic optimization

A good generator use case is one where producing everything up front would be wasteful or less natural than yielding values as they are needed.