Using python logging package, how to insert additional formatting into multiple logger handlers that have their own formatters?

Question

I have a single logger with multiple handlers, which have their own formatters. Now I want to add an indenting feature, with the indent level controlled at runtime. I want messages from all the handlers to get this indent. I've tried to create it as a filter, but found that I seem to be unable to alter the message content. Then I've tried it as a formatter, but I can only have one per handler. How can I add such indentation without explicitly changing the formatters of every handler?
I should mention that one of the formatters I have is a class that adds color to the output. It is not a simple format string.

---

In addition, I am using a config file. Ideally, I'd like to be able to drive this mostly from there. However, I need to modify the state of the indent formatter (e.g. set indent level), but I don't know how to get to that particular formatter as there's no logger.getFormatter("by_name") method.
To clarify, I need to access the specific formatter instance, essentially to adjust the format on the fly. The instance has been created by logging.config from the file. I don't find any accessor methods that would allow me to get the particular formatter given its name.

Answer 1

Here's another, hacky, but simple one. My messages for all handlers always start with a message level string. Just modify those darn strings on every indent change:

# (make a LEVELS dict out of all the logging levels first)    
def indent(self, step = 1):
        "Change the current indent level by the step (use negative to decrease)"
        self._indent_level += step
        if self._indent_level < 0:
            self._indent_level = 0
        self._indent_str = self._indent_str_base * self._indent_level
        for lvl in LEVELS:
            level_name = self._indent_str + LEVELS[lvl]
            logging.addLevelName(lvl, level_name)

(see my other answer for the stuff that surrounds indent function)
Now the indenter can be an independent class without messing with the details of logging process. As long as the message includes the level string, the indent will be there, even if some stuff is coming before it. Not ideal in general, but may work for me.
Anybody has more ideas that work for any msg format?

Answer 2

#!/usr/bin/env python

import logging
from random import randint

log = logging.getLogger("mylog")
log.setLevel(logging.DEBUG)

class MyFormatter(logging.Formatter):
    def __init__(self, fmt):
        logging.Formatter.__init__(self, fmt)

    def format(self, record):
        indent = " " * randint(0, 10) # To show that it works
        msg = logging.Formatter.format(self, record)
        return "\n".join([indent + x for x in msg.split("\n")])

# Log to file
filehandler = logging.FileHandler("indent.txt", "w")
filehandler.setLevel(logging.DEBUG)
filehandler.setFormatter(MyFormatter("%(levelname)-10s %(message)s"))
log.addHandler(filehandler)

# Log to stdout too
streamhandler = logging.StreamHandler()
streamhandler.setLevel(logging.INFO)
streamhandler.setFormatter(MyFormatter("%(message)s"))
log.addHandler(streamhandler)

# Test it
log.debug("Can you show me the dog-kennels, please")
log.info("They could grip it by the husk")
log.warning("That's no ordinary rabbit!")
log.error("Nobody expects the spanish inquisition")
try:
    crunchy_frog()
except:
    log.exception("It's a real frog")

result:

    They could grip it by the husk
    That's no ordinary rabbit!
          Nobody expects the spanish inquisition
         It's a real frog
         Traceback (most recent call last):
           File "./logtest2.py", line 36, in 
             crunchy_frog()
         NameError: name 'crunchy_frog' is not defined

I'm not sure I understand your second question.

Answer 3

Ok, here's one way that gets me ALMOST what I need. Subclass the LogRecord to overwrite getMessage for inserting indent and subclass logger to makeRecord with it:

import logging
import logging.config  

################################################################################
class IndentingLogger(logging.Logger):
    """A Logger subclass to add indent on top of any logger output
    """
    ############################################################################
    def __init__(self, name = 'root', logging_level = logging.NOTSET):
        "Constructor to keep indent persistent"
        logging.Logger.__init__(self, name, logging_level)
        self.indenter = IndentedRecord("", logging.NOTSET, "", 0, None, None, None, None, None)

    ############################################################################
    def makeRecord(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
        return self.indenter.set_record(name, level, fn, lno, msg, args, exc_info, func, extra)


################################################################################
class IndentedRecord(logging.LogRecord):
    """A LogRecord subclass to add indent on top of any logger output
    """
    ######## Class data #########
    DEFAULT_INDENT_STR = '    '

    ############################################################################
    def __init__(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
        "Constructor"
        logging.LogRecord.__init__(self, name, level, fn, lno, msg, args, exc_info, func)
        self._indent_level = 0
        self._indent_str_base = IndentedRecord.DEFAULT_INDENT_STR
        self._indent_str = ""    # cache it

    ############################################################################
    def set_record(self, name, level, fn, lno, msg, args, exc_info, func=None, extra=None):
        "Constructs the base record"
        logging.LogRecord.__init__(self, name, level, fn, lno, msg, args, exc_info, func)
        return self

    ################################################################################
    def getMessage(self):
        "Adds indent on top of the normal getMessage result"

        # Call base class to get the formatted message
        message = logging.LogRecord.getMessage(self)

       # Now insert the indent
        return self._indent_str + message

    ################################################################################
    def indent(self, step = 1):
        "Change the current indent level by the step (use negative to decrease)"
        self._indent_level += step
        if self._indent_level < 0:
            self._indent_level = 0
        self._indent_str = self._indent_str_base * self._indent_level

    ################################################################################
    def set_indent_str(self, chars):
        "Change the current indent string"
        if not isinstance(chars, str):
            raise ValueError("Argument must be a string. Got %s" % chars)
        self._indent_str_base = chars

logging.config.fileConfig("reporter.conf")
logging.setLoggerClass(IndentingLogger)
logger = logging.getLogger('root') # will be wrong logger, if without argument

logger.debug("debug message")
logger.info("info message")
logger.indenter.indent(+1)
logger.warn("Indented? warn message")
logger.indenter.set_indent_str("***")
logger.error("Indented? error message: %s", "Oops, I did it again!")
logger.indenter.indent(+1)
logger.error("Indented? error message: %s", "Oops, I did it again!")
logger.indenter.indent(-1)
logger.critical("No indent; critical message")

The result is (colored in reality):

Debug: debug message
Info: info message
Warning:     Indented? warn message
Error:     Indented? error message: Oops, I did it again!
Error: ******Indented? error message: Oops, I did it again!
Internal Error: ***No indent; critical message

Somehow the log level string still sneaks to the front, so it is not quite what I want. Besides, this is awkward - too much for such a simple feature :(
Better ideas?