CKAN server error

Question

From time to time I am getting a timeout error from my CKAN server. The log file shows the following error message:

TimeoutError: QueuePool limit of size 5 overflow 10 reached.

Any idea what's causing this and how to fix it?

PK

Answer 1

The QueuePool I think refers to the database connections limit. Are you not closing connections in an extension or getting a very large amount of traffic?

Answer 2

This is CKAN's SQLAlchemy running out of connections to the database. You'll see it with when apache sees several simultaneous requests under a bit of load. You can replicate it using something like 'ab' to provide the simultaneous load.

The SQLAlchemy pool of connections is shared amongst threads in the same CKAN process, but each CKAN process has its own pool.

You could increase the numbers of connections in the pool, but remember that postgres has a limit of 100 connections by default. So with multiple processes you'll hit a limit reasonably quickly. Or you can limit the number of CKAN processess. I believe Ian noted a few days ago one CKAN dependency that is not thread-safe, so best stick to just processes for now.

If using Apache, you could run the mod_wsgi in daemon mode and specify limits for the processes/threads. If you're on nginx/uwsgi then this is naturally limits the number of processes, which is better.

The best solution would be to use a separate app to hold the connection pool. That has the advantage of working across CKAN threads. PGPool is an option, but we went for pgbouncer, which I believe is faster and we've been very pleased with it.