How to add large quantities of files to zip in a memory efficient way?

Question

I'm a bit new to optimizing programs for memory efficiency, so this might seem a little rudimentary: If I intend to add hundreds of files (possibly 100mb+) to a zip, which method would be better for keeping system resources available? (And for catching)

The zip_queue variable below is a list of globbed files.

My current method:

with zipfile.ZipFile(zip_name, "a", compression=zipfile.ZIP_DEFLATED) as myzip:
    for file in zip_queue:
        myzip.write(file) 

OR this way:

for file in zip_queue:
    with zipfile.ZipFile(zip_name, "a", compression=zipfile.ZIP_DEFLATED) as myzip:
        myzip.write(file)

I'm guessing the latter would flush the memory each time it appends the file, but would opening and closing the zip each time incur greater memory costs than simply keeping the file open?

Thanks!

Answer 1

I'd do it the first way, simply to avoid closing and re-opening the same output file over and over. I doubt the fine details of exactly how much memory each approach uses matter, and if those details do matter so much, you need to write code in something other than Python.

As a general rule, file access is slower than memory, so I wouldn't worry about memory usage at all here unless you have an actual problem you can demonstrate.