Google indexing cfc's and giving 500 error

Question

So I'm working on a ColdFusion site and this morning we found out that Google is crawling our site and following cfc and getting a 500 error. We are using the cfc with ajax calls so they should not be crawled. How can we fix this?

Answer 1

The only reason I can think of that Google would index your cfc's would be that it is finding links to them in your pages. Remember, the Google bot can also find the links in your JavaScript code. You should be able to create/modify your robots.txt file to tell the search engines to exclude the directory(ies) that contain your cfc's from their indexes.

Sample robots.txt entry:

User-Agent: *
Disallow: /cfc-directory/

The Google bot (but not all search engines) can even support some pattern matching (reference). So you could tell the Google bot not to index any files ending with .cfc by doing this:

User-agent: Googlebot 
Disallow: /*.cfc$

A quick search turned up this similar question. In it @nosilleg mentions that the javascript code generated by ColdFusion's cfajaxproxy includes links to cfc's (in particular to /baseCFC/Statement.cfc. So if you are using that in any of your pages it will also contain links to cfc's.