Python: File Handler Issue: Delete file without leaving .nsf files

Question

I have following method to handle logging in my python program

def createLogger(logger, logLang):
    """
    Setting up logger
    """
    log_format = logging.Formatter("%(asctime)s - %(name)s - %(levelname)s - %(message)s")
    file_handler = logging.FileHandler(filename=(os.path.join(OUT_DIR_LOGS, logLang + '-userdynamics.log')))
    file_handler.setFormatter(log_format)
    logger.setLevel(logging.INFO)
    logger.addHandler(file_handler)

This is a large data collection code base and to avoid quota constrain on remote server, I have implemented following gzip and tar procedure,

def gzipLogs(lang):
    """
    Compressing and tar
    """
    # Compressing logfiles and removing the old logfile
    original_filePath = OUT_DIR_LOGS + "/" +lang + "-userdynamics.log"
    gzip_filePath = OUT_DIR_LOGS + "/" + lang +"-userdynamics.gz"
    with open(original_filePath , 'rb') as original_file:
        with gzip.open(gzip_filePath, 'wb') as zipped_file:
            zipped_file.writelines(original_file)
    os.remove(original_filePath)
    # Compressing language folders that has data
    folder_path = OUT_DIR + "/" + lang
    tar_file = tarfile.open(folder_path + ".tgz", "w:gz")
    # add timestamp to arch file
    tar_file.add(folder_path, arcname = NOW + "_" + lang)
    tar_file.close()
    # delete the original file
    shutil.rmtree(folder_path)

I do my data gather process in a nested for loop and I call the logger as mentioned bellow:

for something in somethings:
    for item in items:
        log = logging.getLogger()
        # Calling the logging configuration function.
        createLogger(log, lang)

Everything works fine but when it get executed, after deleting the file .nsf file residuals get left behind while causing the quota problem to remain as it is.

So I added following code segment to close the logging file handler but with this now I endup getting following error:

Code to close the log file:

unclosed_logs = list(log.handlers)
for uFile in unclosed_logs:
    print uFile
    log.removeHandler(uFile)
    uFile.flush()
    uFile.close()

Above code end up giving me this error:

Traceback (most recent call last):
  File "/somefilepath/SomePythonFile.py", line 529, in <module>
    main()
  File "/somefilepath/SomePythonFile.py", line 521, in main
    gzipLogs(lang)
  File "/somefilepath/SomePythonFile.py", line 61, in gzipLogs
    with gzip.open(gzip_filePath, 'wb') as zipped_file:
AttributeError: GzipFile instance has no attribute '__exit__'

This is how main method looks like with the handler closing code segment:

for something in somethings:
    for item in items:
        log = logging.getLogger()
        # Calling the logging configuration function.
        createLogger(log, lang)
    unclosed_logs = list(log.handlers)
    for uFile in unclosed_logs:
        print uFile
        log.removeHandler(uFile)
        uFile.flush()
        uFile.close()

What am I doing wrong? Am I handling the logger wrong? or am I closing the file too soon?

Answer 1

After some research I realize that server that I was executing the file was running against python 2.6 and in 2.6 GZip module didn't have with open. To answer for this question is either switch to python 2.7 or change the implementation to old fashion file open file in a try catch block.

try:
    inOriginalFile = open(original_filePath, 'rb')
    outGZipFile = gzip.open(gzip_filePath, 'wb')
    try:
        outGZipFile.writelines(inOriginalFile)
    finally:
        outGZipFile.close()
        inOriginalFile.close()
except IOError as e:
    logging.error("Unable to open gzip files, GZIP FAILURE")

This is how I fixed this problem.

Answer 2

There are a number of things which could cause problems:

1. You should only configure logging (e.g. set levels, add handlers) in one place in your program, ideally from the if __name__ == '__main__' clause. You seem not to be doing this. Note that you can use WatchedFileHandler and use an external rotator to rotate your log files - e.g. logrotate provides rotation and compression functionality.
2. Your error relating to __exit__ is nothing to do with logging - it's probably related to a Python version problem. GZipFile only became usable with with in Python 2.7 / 3.2 - in older versions, you will get the error message if you try to use a GZipFile in a with statement.