When is a generator better than a list comprehension?

Category: Python Fundamentals

Answer

Generators produce one item at a time and hold no buffer — perfect for streaming or large pipelines. Lists materialize the whole iterable in memory. Use a generator when the consumer processes items one-at-a-time and the data could be huge.

Concrete examples from the fca project context

Example 1

(x*x for x in range(10**9)) streams, never holds 10^9 ints in memory.

Example 2

def read_lines(f): for line in f: yield line.strip() — works on 100 GB files.

Example 3

Pipeline of generators def a(); def b(); def c(); for x in c(b(a())) — O(1) extra at each stage.

Data flow / flow chart

[1, 2, 3, ..., N] all in RAM
  vs.
  yield 1; yield 2; ... one at a time, O(1)

Takeaway

Generator when one-at-a-time + memory matters. List when you need slicing or len().