Celery Python logging config to log DEBUG only from specified modules

Question

I'm trying to get django celery to log (to console) DEBUG (and higher) events coming from my modules (and not those of libraries). So starting celeryd with:

python hack/manage.py celeryd -E -l DEBUG -c 2

will mean that all DEBUG events are passed to console (which is a handler on my root logger). I'd like to modify this behaviour so that:

• DEBUG events from mymodule (and children) go to console
• DEBUG events from all other modules are ignored (eg: go to the Null Handler)
• all other events are processed as normal (eg: ERROR continues to go to the mail handler)

What is the best way to do this?

Answer 1

I have stubbled with this and there isn't a single "best" way to do this. It's important to keep in mind a few things.

• Celery is run on production as a separate executable. This seems obvious but that means that sharing log files with other stuff (like django) won't work nicely (See CELERYD_HIJACK_ROOT_LOGGER) so don't do it..
• Keeping things running in a similar manner on both production and your development environment is really critical so things don't go awry when you move to procution (See CELERY_ALWAYS_EAGER and use it)
• Celery when run from manage.py will reference your settings.py file for configuration if you use settings

With that in mind here is how I would suggest you set up your logging to meet your needs. First make sure that you have the logging level for your modules appropriately set. See my other post on how to set up logging. Here is a condensed version to meet your needs

LOGGING = {
    'version': 1,
    'disable_existing_loggers': True,
    'formatters': {
        'standard': {
            'format': "[%(asctime)s] %(levelname)s [%(name)s.%(funcName)s:%(lineno)d] %(message)s",
            'datefmt': "%d/%b/%Y %H:%M:%S"
        },
    },
    'handlers': {
        'console': {
            'level': 'DEBUG',
            '()': 'logutils.colorize.ColorizingStreamHandler',
            'formatter': 'standard',
            'stream': sys.stdout
        },
        'mail_admins': {
            'level': 'ERROR',
            'class': 'django.utils.log.AdminEmailHandler',
            'include_html': True,
        },

    },
    'loggers': {
        'django': {
            'handlers': ['mail_admins'],
            'propagate': True,
            'level': 'ERROR',
        },
        'mymodule': {
            'handlers': ['console', 'mail_admins'],
            'propagate': True,
            'level': 'DEBUG'

        },
    }
}

Then run it like this.

./manage.py celeryd --event --beat --settings=dev

What this does is sets two different handlers. The first one is the console handler which reports out anything DEBUG and above. The second is the mail handler which only looks at ERRORS and above. Next is what gets reported from the loggers. All django.* loggers will get reported to the any ERROR level handler (mail_admins). And any of our modules get push DEBUG and above to both.

I think this is what you are after.