How to do HTTP request logging as a means of troubleshooting login errors

Question 1

I think the best option is to use a suite of Logstash (event collecting) + Elasticsearch (event storage) + Kibana (analytics). All three are really good opensource projects with a lot of documentation and very active communities.

And if you need commercial support for any you can request help from: http://www.elasticsearch.org/

Logstash its flexible enough to allow you parse many log file formats out of the box. Moreover, storing all your logs on elastic search will allow you to create custom queries, reports and stuff.

You can check a kibana demo on: http://demo.kibana.org/

Links: http://www.elasticsearch.org/overview/kibana/ http://logstash.net/

Question 2

As a temporary thing this probably does not require any sophisticated solution.

I successfully used this quick and dirty Django middleware for quite similar purpose - https://gist.github.com/Suor/7870909

Question 3

Have you given Sentry a look? https://getsentry.com/welcome/ and https://github.com/getsentry/raven-python. It might be overkill for what your issue is though. Why not just implement more verbose logging in your authentication methods and setup a separate logger just for authentication failures?

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': ('[%(levelname)s] - %(asctime)s - %(module)s: '
                       '%(process)d %(thread)d %(message)s')
        },
    },
    'handlers': {
        'auth': {
            'level': 'DEBUG',
            'class': 'logging.handlers.RotatingFileHandler',
            'formatter': 'verbose',
            'filename': os.path.join(someloglocation, 'auth.log'),
            'maxBytes': 1024*1024*3,
            'backupCount': 10,
        },
    },
    'loggers': {
        'auth': {
            'handlers': ['console', 'auth'],
            'level': 'DEBUG',
            'propagate': True,
        },
    },
}

Seems like it'd be easier to change the log level from DEBUG to WARN or vice-versa than to implement some logging setup that's overkill for what your needs are. As far as searching by ip or username goes:

cat auth.log | grep <ip address> | egrep -v ' (200|301|302|304) '
cat auth.log | grep <username> | egrep -v ' (200|301|302|304) '

That, however, requires that you're logging all of that information in the first place. I don't think there's a one-size fits all model for this because with middleware you're going to log everything for every request, not just your authentication views. By logging more verbosely where you're authenticating you'll actually get valid results.

Question 4

There are a couple of ways this can be handled within django

Add a middleware to the django app that logs the request and all data you need from the request i.e request.POST if it is a POST and request.GET if it is a GET.

In a file called middleware.py

import logging
logger = logging.getLogger('app')

class RequestLoggingMiddleware:
    def process_request(self, request):
        logger.debug(request)
        logger.debug('Logged Request')
        return None

In your settings.py, add middleware.RequestLoggingMiddleware to the MIDDLEWARE_CLASSES.

Somethings related to this are listed on Is there a Django middleware/plugin that logs all my requests in a organized fashion? and https://github.com/kylef/django-request

The other option is to add a log handler that logs requests that error out. The logging level can be changed to debug to log all requests

From official docs

django.request Log messages related to the handling of requests. 5XX responses are raised as ERROR messages; 4XX responses are raised as WARNING messages.

Messages to this logger have the following extra context:

status_code: The HTTP response code associated with the request. request: The request object that generated the logging message.

Add the below handler to your log dict config file.

'django.request':
        {
            'handlers':
                ['mail_admins', 'console', 'file'],
            'level':
                'ERROR',
            'propagate':
                False,
        },

Once you have setup logging requests, and have your logs being collected somewhere, there are different ways to analyze them. I use Logentries which collects my logs and provides an interface where I can filter by time and do a grep like search. Sometimes when this is not adequate, I download the logs as a tar and use a version of Splunk running locally that has better search tools. But as long as you can filter by time and find the appropriate request logs, you should be able to debug what is happening.