Several `with`s in `try`s
-
03-07-2021 - |
Question
I have several possible files which could hold my data; they can be compressed in different ways, so to open them I need to use file()
, gzip.GzipFile()
and other which also return a file object (supporting the with
interface).
I want to try each of them until one succeeds in opening, so I could do something like
try:
with gzip.GzipFile(fn + '.gz') as f:
result = process(f)
except (IOError, MaybeSomeGzipExceptions):
try:
with xCompressLib.xCompressFile(fn + '.x') as f:
result = process(f)
except (IOError, MaybeSomeXCompressExceptions):
try:
with file(fn) as f:
result = process(f)
except IOError:
result = "some default value"
which obviously isn't feasible in case I have dozens of possible compression variants. (The nesting will get deeper and deeper, the code always looking very much alike.)
Is there a nicer way to spell this out?
EDIT: If possible I'd like to have the process(f)
out of the try/except as well to avoid accidental catching of exceptions raised in the process(f)
.
Solution
I'd write a custom context manager:
from contextlib import contextmanager
filetypes = [('.gz', gzip.GzipFile, (IOError, MaybeSomeGzipExceptions)),
('.x', xCompressLib.xCompressFile, (IOError, MaybeSomeXCompressExceptions))]
@contextmanager
def open_compressed(fn):
f = None
try:
for ext, cls, exs in filetypes:
try:
f = cls(fn + ext)
except exs:
pass
else:
break
yield f
finally:
if f is not None:
f.close()
with open_compressed(fn) as f:
result = "some default value" if f is None else process(f)
Or possibly just a function that returns a context manager:
filetypes = [('.gz', gzip.GzipFile, (IOError, MaybeSomeGzipExceptions)),
('.x', xCompressLib.xCompressFile, (IOError, MaybeSomeXCompressExceptions))]
class UnknownCompressionFormat(Exception):
pass
def open_compressed(fn):
for ext, cls, exs in filetypes:
try:
return cls(fn + ext)
except exs:
pass
raise UnknownCompressionFormat
try:
with open_compressed(fn) as f:
result = process(f)
except UnknownCompressionFormat:
result = "some default value"
OTHER TIPS
Yea, you could put all your variants through a list and try them until one of them works, thus un-nesting your code:
def process_gzip(fn):
with gzip.GzipFile(fn + '.gz') as f:
return process(f)
def process_xlib(fn):
with xCompressLib.xCompressFile(fn + '.x') as f:
return process(f)
def process_builtin(fn):
with file(fn) as f:
return process(f)
process_funcs = [process_gzip, process_xlib, process_builtin]
#processing code:
for process_f in process_funcs:
try:
result = process_f(fn)
break
except IOError:
#error reading the file, keep going
continue
except:
#processing error, re-raise the exception
raise
Or, to reduce amount of code you could make a process_func factory, since they all have the same form:
def make_process_func(constructor, filename_transform):
with constructor(filename_transform) as f:
return process(f)
process_funcs = [
make_process_func(gzip.GzipFile, lambda fn: fn + '.gz'),
make_process_func(xCompressLib.xCompressFile, lambda fn: fn + '.x'),
make_process_func(file, lambda fn: fn),
]
Would this work:
extensions = [('.gz', gzip.GzipFile, (IOError, MaybeSomeGzipExceptions)),
('.x', xCompressLib.xCompressFile, (IOError, MaybeSomeXCompressExceptions))] # and other such entries
processed = False
for ext, (compressor, errors) in extensions.iteritems():
try:
with compressor(fn+ext) as f:
try:
result = process(f)
processed = True
break
except:
raise
except errors:
pass
if not processed:
result = "some default value"
Hope that helps