PYTHON / ITERATORS, GENERATORS, AND COMPREHENSIONS
Memory profile: list versus generator
Measure and explain why a million-item list costs megabytes while an equivalent generator stays a few hundred bytes, and when the list still wins.
What you will learn
- Predict a CPython list's own size as 56 + 8*n bytes, with the elements counted separately
- Use tracemalloc's peak value, not sys.getsizeof, to compare list and generator workloads
- Explain why a generator object's size is unrelated to how many items it will yield
- Spot the consumers (len, indexing, a second pass) that force materialization anyway
Understanding Memory profile: list versus generator
A list is a contiguous array of pointers. On 64-bit CPython the list object itself costs 56 bytes (including its GC header) plus 8 bytes per slot, and every element it points at is a separate object with its own header, so a million small ints add roughly 28 MB on top of the 8 MB pointer array. A generator stores something completely different: a reference to its code, one execution frame holding a couple of locals, and a state flag. That footprint is fixed at a few hundred bytes whether the generator will produce ten items or ten billion, because at any instant only one item exists.
sys.getsizeof answers a narrow question: how many bytes does this one object occupy, not counting anything it refers to. That is why it reports 8000056 for list(range(1_000_000)) while ignoring the ints inside, and why it reports a couple of hundred bytes for a generator that could yield a terabyte of data. For an honest comparison you must measure allocations as the work happens: call tracemalloc.start(), run the variant, then read the second value from get_traced_memory(). That peak, the high-water mark, is the number that decides whether your process survives.
The lower peak is not free. A generator supports exactly one forward pass, so len() raises TypeError, indexing is impossible, and a second loop sees nothing at all. Resuming a frame per item also costs CPU, so for a few thousand elements a list comprehension is usually the faster choice. Pick a generator when the data is large or streamed and each item is touched once; pick a list when you need length, random access, or repeated passes, and remember that a single downstream list(), sorted(), or len() puts the full memory cost straight back.
import sys
n = 1_000_000
big_list = list(range(n)) # a million int pointers exist right now
big_gen = (i for i in range(n)) # nothing computed yet
print("list object:", sys.getsizeof(big_list), "bytes")
print("56 + 8 * n:", 56 + 8 * n)
print("generator object under 1 KB:", sys.getsizeof(big_gen) < 1024)
tiny_gen = (i for i in range(5))
print("generator size grows with n:", sys.getsizeof(tiny_gen) != sys.getsizeof(big_gen))
print("sum via list:", sum(big_list))
print("sum via generator:", sum(big_gen))
print("second pass over generator:", sum(big_gen))A list pays memory proportional to the number of items it holds, while a generator pays for one item plus a fixed-size frame no matter how many items it will ever produce.
Worked examples
Peak allocation with tracemalloc
Compares the real high-water memory of summing squares through a list comprehension versus a generator expression.
import tracemalloc
def peak_bytes(build):
tracemalloc.start()
build()
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
return peak
n = 200_000
list_peak = peak_bytes(lambda: sum([i * i for i in range(n)]))
gen_peak = peak_bytes(lambda: sum(i * i for i in range(n)))
print("list peak over 4 MB:", list_peak > 4_000_000)
print("generator peak under 50 KB:", gen_peak < 50_000)
print("list peak at least 20x generator peak:", list_peak > 20 * gen_peak)Example explained
Line 1get_traced_memory() returns (current, peak); the second value is the high-water mark since start(), which is what a list spike actually costs.
Line 2The list version keeps 200,000 int objects plus a ~1.6 MB pointer array alive until sum() finishes, landing near 8 MB on CPython.
Line 3The generator version holds one square, one range, and the running total, so its peak is a few kilobytes.
Line 4tracemalloc.stop() resets tracing so the second measurement starts from zero instead of inheriting the first peak.
What the memory saving costs you
Shows the capabilities a generator gives up, and that restoring them restores the full memory cost.
squares = (i * i for i in range(5))
try:
len(squares)
except TypeError as exc:
print("len:", exc)
print("first pass:", list(squares))
print("second pass:", list(squares))
materialized = list(i * i for i in range(5))
print("materialized:", materialized, "len", len(materialized), "item 2", materialized[2])Example explained
Line 1len(squares) fails because a generator has no __len__: the count is unknown until the items have been produced and thrown away.
Line 2The first list(squares) drains the generator and returns all five values.
Line 3The second call returns [] because an exhausted generator keeps raising StopIteration rather than restarting.
Line 4Wrapping the generator expression in list() buys back len and indexing at exactly the memory price you were trying to avoid.
How many objects ever exist
Counts the items actually created when a search stops early, versus when the same generator is materialized first.
produced = 0
def squares(n):
global produced
for i in range(n):
produced += 1
yield i * i
first = next(x for x in squares(200_000) if x > 50)
print("lazy:", first, "items created:", produced)
produced = 0
first = next(x for x in list(squares(200_000)) if x > 50)
print("eager:", first, "items created:", produced)Example explained
Line 1The counter increments once per yield, so it records exactly how many objects were ever built.
Line 2The lazy search stops at i = 8 (8 * 8 = 64 is the first square above 50), so only 9 ints were created and only one was held at a time.
Line 3Wrapping the same generator in list() drains all 200,000 items before the search even starts.
Line 4Identical answer, more than 20,000x the objects created: peak memory follows how much you materialize, not how much you inspect.
A generator that saves nothing
Demonstrates that a generator whose body accumulates state has the same footprint as a list.
import tracemalloc
def running_groups(n):
seen = {}
for i in range(n):
seen[i] = i * i
yield len(seen)
tracemalloc.start()
last = 0
for value in running_groups(50_000):
last = value
_, peak = tracemalloc.get_traced_memory()
tracemalloc.stop()
print("last value:", last)
print("peak over 1 MB:", peak > 1_000_000)Example explained
Line 1seen lives in the generator's frame and survives every suspension, so it grows to 50,000 entries.
Line 2Each yield hands back one small int, yet the peak is megabytes because the dict is still fully resident.
Line 3The rule is about the state a generator keeps between yields, not about the yield keyword itself.
Important notes
The exact byte counts here are CPython 64-bit values (empty list 56 bytes including its 16-byte GC header, 8 bytes per slot); treat them as relative comparisons, not portable constants.
A generator only costs O(1) memory if its body keeps O(1) state between yields, and it never makes data smaller: it just avoids holding all of it at once.
Common mistakes
Comparing sys.getsizeof(list_of_a_million) with sys.getsizeof(generator) and reporting the ratio as the memory saving: getsizeof is shallow, so it ignores the element objects on one side and the unproduced values on the other, giving a number that means nothing.
Reading get_traced_memory()[0] (current) instead of [1] (peak), often after the list has already been freed, which shows almost no difference and leads to the conclusion that generators do not help.
Feeding the memory-efficient generator into list(), sorted(), or len() further down the pipeline, which rebuilds the whole sequence in memory and adds per-item resume overhead on top.
Iterating a generator twice to check the result, getting an empty second pass, and 'fixing' it by materializing the generator, silently undoing the optimization.
Try it yourself
Change, predict, then run
Write sum_list(n) using a list comprehension and sum_gen(n) using a generator expression, both returning the sum of i*i for i below 500_000, then measure each with tracemalloc and print peak_list // peak_gen.
Open the Python workspaceCheck your understanding
data is a list of 1,000,000 floats. You change total = sum([f(x) for x in data]) into total = sum(f(x) for x in data). What happens to peak memory?
- Peak drops to a few hundred bytes, because generators never allocate the values they yield.
- Nothing changes, because sum() materializes the generator into a list internally.
- Peak drops by roughly the size of the temporary result list, but the 1,000,000-element data list is still fully resident.
- Peak rises, because each resumed generator frame is larger than a list slot.
Show answer
sum() only calls __next__ and adds each value to a running total, so the intermediate list of results never exists and disappears from the peak; but data itself was already materialized and untouched, so memory cannot fall to a constant. Option 1 is tempting because sum() often appears with brackets, yet sum() requires only an iterable, never a length or indexing, so it does not build a list.