Question

Looking through my search logs from time to time, I notice that by far the biggest user of my search engine is the google-bot. What gives? Is it looking for content that might not be directly accessible through navigation? If so, how does it know which words and phrases to look for (they're surprisingly relevant). Does it check the most popular keywords on the site? I know I seem to be answering my own question here, but this is really only working it out from first principles. I'd like to hear from someone who knows what they're talking about (i.e. not me).

Was it helpful?

Solution

Google will use words that occur on your site in search boxes to try to find pages that it can't otherwise.

Google says that for the past few months, it has been filling in forms on a "small number" of "high-quality" web sites to get back information. What words has it been entering into those forms? Words automatically selected that occur on the site, with check boxes and drop-down menus also being selected. http://searchengineland.com/google-now-fills-out-forms-crawls-results-13760

OTHER TIPS

If your search form's method is get instead of post, each search has its own url, and people might be posting those urls elsewhere. Or if you have a (possibly inadvertently) publicly accessible webstats page that listed those urls, that's another common way for search engines to stumble upon your internal search urls. A third way I've seen is sites that list recent searches on their pages, but this is more intentional. "MySQL Performance Blog" does this to an annoying extent, so any search of their site from google yields hundreds of pages of similar searches, even if none of them found what they were looking for.

Edit: Looks like it does on occasion, but only GET forms: http://googlewebmastercentral.blogspot.com/2008/04/crawling-through-html-forms.html

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top