Does the crawl frequency of GoogleBot and MSNBot increase with better response time?

Question

We have network in/out alarms set on our AWS EC2 instances. While the instance was running hot (slower response times - CPU hovering around 85%), I noticed our network in/out was below 2.0E7 bytes.

When we upgraded and shifted things around, and brought our CPU down to around 30%, it seems that we are now getting crawled more frequently.

Whenever we have modified our instance types, I have noticed a few significant short-term spikes. What I suspect is that the crawlers do something like this, detecting response times, thereby dictating their crawl frequencies.

I do realize that I can set the crawl rate in robots.txt - but I don't have any limitation set in there.

Of course there could be other factors involved, but I wanted to know if this makes sense?

Answer 1

Google's documentation doesn't say specifically that they check your response time, but my experience with Googlebot and with other crawlers indicates that it does indeed take your site's response time into effect. I know that my crawler did. I based my crawl rate on the response time of the site I was crawling.

See

• More control of Googlebot's crawl rate
• Changing Google's crawl rate
• search on "googlebot crawl rate" for a lot of info.