Efficient storing a big table in Django cache

Question

I use Django with jqGrid and loading pages via AJAX. At times, queries are very complex, and page loading is ver slow, for far pages is much slower (which is to be expected, the results often exceed 100k objects). I thought that result caching will solve the problem, adding some time to the loading of the first page, but then strongly accelerating the loading of subsequent pages.

Instead, it made the loading of the first page incredibly slow and even subsequent pages take a lot of time (11s on a standard PC). I'm using locmem cache backend. Any ideas? I tried, for a comparison, storing results in global dictionary and that was MUCH better (subsequent pages take only 1s), but I've heard that it's not a safe way.

Any ideas?

Answer 1

You could look at warming your cache. This could be done manually, or using a queuing framework like celery to have the caching of subsequent pages or querysets happen in the background after another page load.

Have a look at johnny-cache, which does transparent queryset caching. This may (I repeat, may) solve all of your problems.