Application response time increase after flushing Redis object cache

Question

So, today we attempted to migrate object cache from "local file storage" to Redis for our store on Magento EE 1.14.2.4.

After placing the connection details in local.xml and manually clearing var/cache/ on all front-end servers, everything seem to be working well as application performance stayed steady and we could see reads/writes on the Redis DB.

About 40 minutes later, we attempted to flush cache from the admin using the System > Cache Management > Flush Magento Cache button.

From this time on, we saw our application response time double and a large increase in writes to Redis DB. This was not just a brief increase as cache was "re-warmed", response time and Redis DB writes stayed high for 20 minutes until we decided to switch back to local file storage for object cache.

During the period of increased application response time, we manually flushed Redis, but it did not solve the issue.

Can anyone shed any light on why we might have seen this behavior?

Here is a visualization of app response time, showing the effect of switching to Redis for object cache and flushing cache from the admin:

Answer 1

We were able to solve this issue via a patch from Magento.

Here is an explanation of the issue:

The Magento config is cached on the first request of the site; following a cache flush. During the process of the config being cached, a lock is put in place, telling subsequent requests of the site bypass cache and load the config dynamically.

Subsequent requests of the site (while the config is being cached) also attempt to cache the config, leaving the lock in place.

When cache is flushed under a high load, rapid requests of the site do not allow any single "uncached hit" to finish caching the config before another request starts the caching process over again.

The patch adds the following functionality:

If the "config cache lock" is present, the request waits .5 sec, then tries again up to 10 times.

What this effectively does is, give the first request (after the cache has been cleared) 5 seconds to cache the config before any subsequent requests will attempt to do so.

---

I haven't seen this patch published anywhere by Magento, so I added to my GitHub in case it can help someone else: https://github.com/timciep/magento-tools/blob/master/Patches/SUPEE-9341_EE_1.14.0.0_v1/PATCH_SUPEE-9341_EE_1.14.0.0_v1.sh