Magento Automatic Caching Insight

Question

We are running Magento EE 1.11 with memcache. 2GB per memcahce server, 4GB total. We have about 240k products.

• Available ram: 6GB
• Cores: 16
• Threads: 32

Here is the deal, more new products are added and changes to products occur on a daily basis and of course each time a new product is added/modified in the back-end the cache becomes invalidated, specifically the 'Full Page Cache'.

When Magentos Auto Cache Generation is enabled it takes about 2 days to build up the cache, with 8 threads allocated to the crawler. After 2 days the memcache floats around ~2GB separated between the two ram disks.

The problem is, when products are modified on a daily basis the cache becomes invalidated and as soon as 'Full Page Cache' is refreshed those 2GB of cache are back to square one, 0, and the viscous cycle of Magentos Auto cache starts again. Now, not only is the cache back to 0, but the CPU usage spikes to 90% and the website turns into a 5-10+ seconds waiting game and you can just forget about trying to request a product with 100+ variations, if it's not cached it would take minutes to load the first time, it's ridiculous.

So, my questions to the community.

• Is there a way for Magento to automatically "update" the cache if invalidated, without rebuilding the whole cache and starting from 0? I know when the cache is invalidated that Magento knows somethings has changed, but not exactly where in the cache (correct me if I'm wrong). Are there modules/configurations out there to bypass rebuilding the whole cache?

On the side note, we are using Tiny Bricks LightSpeed module.

• Can Magentos Automatic Cache Generation be time controlled with a cronjob? Say, to start crawling at 10pm - 6am.

• What would be the best way to approach this situation?, As you understand rebuilding a cache that's in gigabytes everyday is just not acceptable.

Answer 1

You don't have nearly enough RAM

We have about 240k products
Available ram: 6GB
Threads: 32

You do not have nearly enough RAM for the amount of products you have. As a rule of thumb, we recommend at least 2-4GB RAM per logical core.

If you map out your possible memory usage:

• 64 PHP Threads with a max_memory of ~768MB = 24GB
• 240,000 Products will likely mean about 15GB of InnoDB table space
• 64 PHP Threads will warrant around of 128 MySQL connections, typically this comes at a cost of about 200MB per connection minimum
• Backend storage for 240,000 products in Redis AND lzf compressed - will still consume about 6GB of RAM

So the total so far is 70GB of committed RAM - we've not even mentioned the OS etc.

Your hardware is dreadfully underspecified. I would suggest reading over this Magento server set up article for a bit of a feeler on how to progress.

Memcached doesn't support cache tags

If you're using Memcached (not a problem, its very high performance), then you are either storing cache tags or not. If you don't have a slow_backend defined - then you're not storing tags, which basically means your cache cannot discriminate between any of the different cache types - so you won't be able to flush them independently.

Have a read over this, http://www.sonassi.com/knowledge-base/magento-kb/what-is-memcache-actually-caching-in-magento/

We would strongly suggest switching over to Redis. It has its quirks and does need significant fine-tuning for larger stores. But as a whole will perform slightly better than Memcached with the real benefit of cache-tag support.

404's and FPC

FPC has a real problem, in-fact, all caching engines have a problem with 404s. The reason being, any old URLs still being crawled or linked to will land on a page that has to iterate through the entire core_url_rewrite table, try and find a match against all defined routers and namespaces before finally giving up and loading a 404.

Then caching a resource which has no value and will consume space in your cache storage. You'll probably find a huge proportion of your Memcached storage is actually being eaten by 404 content.

With large catalogues (240k products), you're certainly going to have your fair share of product turnover, and thus, changes in URLs and subsequently, many 404's.

FPC Invalidate vs. Clean

At the moment - and by default - FPC's behaviour is to clean the cache on changes, rather than merely invalidate the cache entry. We wrote an extension to alter this behaviour for an EE store to do exactly what you required.

Here's a quick patch to give you an idea of how to solve your issue.

app/code/core/Enterprise/PageCache/etc/config.xml
index 6a56a80..85ebc92 100644
--- app/code/core/Enterprise/PageCache/etc/config.xml
+++ app/code/core/Enterprise/PageCache/etc/config.xml
@@ -139,7 +139,7 @@
             <observers>
                 <enterprise_pagecache>
                     <class>enterprise_pagecache/observer</class>
-                    <method>cleanCache</method>
+                    <method>invalidateCache</method>
                 </enterprise_pagecache>
             </observers>
         </catalogrule_after_apply>

Don't run a crawler

If you've got decent enough footfall - we don't advise running the crawl tool, it generates unnecessary load. People/bots/crawlers browsing the site should keep the cache primed.

But to answer your question, if you look in the configuration file mentioned above - you see the cron schedule that has been defined for the crawl browsing window.

If you can afford stale content

And ultimately, if you have enough RAM. You could well benefit from increasing the TTL of content stored in FPC - to keep your cached data alive for longer.

In the <full_page_cache> tag in your ./app/etc/local.xml just define

<lifetimelimit>86400</lifetimelimit>

The lifetime is defined in seconds. You need to strike a balance between content freshness, performance and the amount of storage space you actually have available.

Why are you using a 3rd party caching extension with EE

You're paying a premium for FPC - which pains me to say, is very good. So why are you running 3rd party alternatives over the top. Remove it.

Put it this way. If your car was running badly - would you just add another engine in the boot to compensate; or just fix the engine already in there?

Answer 2

We understand you very well! We add new or change our products every day and modify static blocks as well. So we were full up with invalidated Magento cache and this constant going to System -> Cache Management. We hated the necessity to refresh the invalidated cache types manually.

Then we installed new Magento Optimizer extension. This module automates this process. It refreshes invalidated/all cache types or flushes Magento cache storage at specified frequency. It was a real relief for all our team!

One more great feature of this extension is that it cleans session files and error reports that are older than X days. Everybody knows that the size of var/session and var/report directories can grow into gigabytes and the number of these files can exceed hundred. So now to slow down the website performance, we set Magento Optimizer once and forgot about periodical refreshing of these directories.

Magento is known for merging an abandoned cart with the current one when customers try to login. From customers’ experience and loyalty point of view, it is destructively. The Magento Optimizer module automatically deletes abandoned carts older that X days. You can use this feature at sales time as well and limit the time for abandoned carts existing.