zen of Python vs with statement - philosophical pondering

Question

I don't intend to simply waste your time, but: has it occurred to you too, while using Python's with statement that it really is contrary to the 5th line of "The Zen of Python" that goes "Flat is better than nested"? Can any enlightened Python guru share me some of their insights on this?

(I always find that one more level of indentation pops up in my code every time I use with instead of f.close()... and it's not like I'm not gonna use try: ... finally: ... anyways and thus the benefits of with still elude me, even as I grow to like and understand Python more and more...)

---

@glglgl (sorry, I can't find a way to write code in comments): yes, but if you go the with way, your code becomes:

try:
    with file(...) as f:
        ...
except IOError:
    ...

...and using just with without the try is what people end up doing in the type of hacky "one use" code where they use f.close() instead of with anyways (which is bad because the file may not be closed if an exception is thrown before their f.close()), so for "hacky" code people just don't use with because, I don't know, I guess they just find it too "fancy" and for well structured code it doesn't bring any benefits anyways, so it seems to me there's no real world use case left for it... that was my pondering about really.

Answer 1

Yes, The Zen of Python states "Flat is better than nested", however it is not the only characteristic we care about; it also states "Simple is better than complex". The beauty of with is that it actually adheres to both of those principles as I will explain below.

Any time you find yourself in philosophical pondering about a feature in Python it's probably worth looking up the Python Enhancement Proposals (PEPs) to read about the motivation behind the feature. In this case PEP 343 -- The "with" Statement says it up front in the abstract:

This PEP adds a new statement "with" to the Python language to make it possible to factor out standard uses of try/finally statements.

Factoring out try/finally statements makes the code simpler and more readable.

PEP 343 goes deeper than providing some simplistic syntactic sugar, however. It establishes a context manager protocol:

The expression immediately following the with keyword in the statement is a "context expression" as that expression provides the main clue as to the runtime environment the context manager establishes for the duration of the statement body.

Using the context manager protocol, API writers can help hide complexity and ensure correct acquisition/release of resources in a multi-threaded context.

But the real beauty of the with statement is shown in Example 12 of PEP 343 which explains that:

A "nested" context manager that automatically nests the supplied contexts from left-to-right to avoid excessive indentation.

Using the nested() context manager you can take code that looks like this:

with a as x:
    with b as y:
        with c as z:
            # Perform operation

and turn it into this:

with nested(a, b, c) as (x, y, z):
             # Perform operation

Note that nested() was introduced in Python 2.5, but as of version 2.7 it is deprecated in favor of this multiple context manager syntactic form:

with a as x, b as y, c as z:
             # Perform operation

Clearly not only is this simpler and more readable, but it is much more flat than nested. Thus, using with is following the path of 無爲 :)

UPDATE: In response to comments on Simeon Visser's answer here is an example of when you might use multiple context managers to open more than one file at once, when you want to zip the contents of two (or more) files together such that if opening one of the files fails it will make the whole thing fail and properly close each file that was opened:

from itertools import izip
with open("/etc/passwd") as a, open("/etc/group") as b, open("/etc/shadow") as c:
    for lines in izip(a,b,c):
        print map(lambda x: x.split(':')[0], lines)

Run this example twice; once as a root and once as normal user. Presuming you save this file as ziptogether.py first try invoking it as root with sudo python ziptogether.py and it will succeed, but invoking it as a normal user with python ziptogether.py will fail because you don't have permissions to read /etc/shadow. When it fails the context manager will ensure that the files that were successfully opened before the failure are properly closed when execution moves outside the scope of the with statement.

Answer 2

Note that the Zen of Python also says:

Simple is better than complex.

Complex is better than complicated.

and

Readability counts.

Using a context manager in the with statement provides multiple things:

• correct behaviour as the file is always closed
• readability (with open(..) as f is quite understandable)

You can't point at one item in the Zen of Python and argue that all Python code must satisfy all items at all times. For example, if the minimum indentation level to solve a particular problem in a readable and correct way is four, then so be it: if an indentation level of three makes the code less readable then just leave the code alone (four is good).

Answer 3

You mention it already: It is cleaner to do

f = file(...)
try:
    # do work on file
finally:
    f.close()

than just closing after the file operations - which would not be reached if an exception occurs.

If you compare the try/finally to with, you have the same level of indentation, so you don't lose anything. If, however, you do exception handling, you have one more level of indentation, which is indeed against the said Zen point.

OTOH, with encapsulates things and makes using them easier and more readable, which are other Zen aspects.

It seems impossible to me to always follow every Zen aspect exactly; sometimes you have to weigh one against the other. In this case, you "lose" one level of indentation, but you get a better readability and maintainability. The latter seems to be an advantage to me.

Answer 4

"Flat is better than nested"

Well, what is flat?

import thirdparty
print "I Like Pie!"

vs

import org.example.thirdparty.something
System.out.println("I Like Cake")

etc...

The Zen of Python does not simply enforce an indentation limit on your code. It encourages you to write readable (and thus, better) code. If your with statement is in a function which is only accessible by 3 layers of objects (etc, one.two.three.func()), Then it's a problem.

Otherwise, three indention levels are just as good a number as any.

Answer 5

The reason to prefer the with is that you do not need to pair manually the related operations (like open(...)/.close(); but the with construct is more general -- not only for working with files). This is important namely in cases when the second operation may not be executed because of the reasons that are not clearly visible from the source code. You are telling the machine take care of it for me, and the machine is better in the case than human. This way you get rid the group of nasty errors that may be difficult to find.

By the way, you should use open(...) instead of the file(...). Python 3 knows nothing about the file(...) and you will otherwise have to fix your code later.