Testing joining list with a 60MB string with `timeit` causes MemoryError

Question

My test appends creates a list containing a 60MB string and a 5-byte string. This list is then joined with join():

import timeit
setup_str = 'str_5byte = "\xfa\xea\x02\x02\x02"; L = [str_5byte]; str_60mb = str_5byte * 12000000'
t = timeit.Timer('L.append(str_60mb); str_long = "".join(L)', setup=setup_str)
t.timeit(100)

Returns this exception:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Python25\lib\timeit.py", line 161, in timeit
    timing = self.inner(it, self.timer)
  File "<timeit-src>", line 6, in inner
MemoryError

I assume that after every execution the variables are deleted and garbage-collected, so why am I running out of memory? Running the test with 8 executions is OK, but higher than that and I get this error.

Answer 1

Yes, with t.timeit the setup statement is executed only once and then the test statement multiple times. This means the list L persists and grows every iteration, resulting obviously in your system running out of memory.

Try min(t.repeat(repeat=100, number=1)) to execute the setup each time before the evaluation of the test statement.

Here's the docs if you need more info.

Answer 2

on a quick experiment, I'm pretty sure setup does exactly the opposite of what you expect - it's run before every call. So you get an extra 60 mb each time through, which does not get collected. When I move that setup code directly into the test code, I am able to run.