Catch "Out of memory error" from python logging

Question

I am running a python scripts which downloads data and processes it. I am also logging some key information. My question is that how will I catch an out of memory exception if thrown by logging, as logging writes to a file. Do I have to put all logging calls within a try and except?

Answer 1

You can create your own logging class that derives from logging but calls log within a try: ... expect: clause.

Answer 2

Instead of catching exception when it's already too late. You can monitor resource usage or specify limits.

Do you mean RAM or disk space?

For disk space you can limit the size of logs, using rotating logs could be useful, where you can specify max size of the logs. See RotatingFileHandler as well as other log handlers. http://docs.python.org/2/library/logging.handlers.html

On Linux you can monitor amount of memory used by your process via resource module.

import resource
print 'Process uses:', resource.getrusage(resource.RUSAGE_SELF).ru_maxrss, "kb"