What is the common/standard practice for Python logging with multiple modules?

Question

Let's say I have a script which relies on two modules:

• Main.py
• Module1.py
• Module2.py

Only Main.py will be executed, but in the execution of Main, there are several calls to methods within Module1.py and Module2.py.

Is it more common to...

1) Log everything into a log for Main.py?

or

2) Log everything into a log for each action's respective module?

(tl;dr if you understand my question, then there's no need to read on. Otherwise, here's an example):

---

Examples (using very abbreviated code):

Case #1 has only one single log file, let's call it Main.log. Assume that the logger name is not hardcoded (it is in this example though), so that should I make AnotherMain.py for example, all of the log entries will go into AnotherMain.log

Main.py:

some_value = Module1.get_value("abc")
some_other_value = Module2.get_value("def")
some_result = some_value + some_other_value
log("main", "some_result evaluated successfully")

Module1.py:

def get_value(key):
   ...
   log("main", "returning value for key from Module1")

Module2.py:

def get_value(key):
   ...
   log("main", "returning value for key from Module2")

This would result in Main.log reading:

some_result evaluated successfully
returning value for key from Module1
returning value for key from Module2

Case #2 has three log files, one for each module: Main.log, Module1.log, Module2.log.

Main.py:

some_value = Module1.get_value("abc")
some_other_value = Module2.get_value("def")
some_result = some_value + some_other_value
log("main", "some_result evaluated successfully")

Module1.py:

def get_value(key):
   ...
   log("Module1", "returning value for key")

Module2.py:

def get_value(key):
   ...
   log("Module2", "returning value for key")

This would result in three logs, each with one line saying only what happened within that module.

I'm not asking, "which do you prefer," but rather, which is more standard and common? Which will leave my successors happier to handle and maintain once I'm gone?

Answer 1

Best practice is to use the logging library, and leave everything to that library.

Each module uses its own logger:

import logging

log = logging.getLogger(__name__)

# ...
log.warn('Do not fnord the frazzle, faf the frobnir instead')

and configure logging in the main module; you can configure one log file, or direct output from specific modules to specific files if so desired.

By default loggers propagate everything on to the root logger, but using separate loggers per module gives you flexibility in the future when you want to set up more complex logging configurations.

You can adjust formatting and direct logging to other backends as needed. The formatter has a lot of options to adjust the generated message, make sure to include the logger name. The configuration options for formatting, filtering and handling are almost endless.

The point is that how many log files you have and what goes into those files is a matter of configuration. Adjust the configuration as your needs change. Start off with something simple; log everything to one file, and progress from there. By giving each module their own logging object, you can configure each logger separately, or stick to the default and leave everything to propagate to the root handler.