preventing google from indexing couchdb design document

Question

I created a web application based on couchdb. It is hosted on cloudant.com (thanks guys, you from cloduant are wonderful!).

The name of the app is R.I.D.E.::Research Infrastructure Database for EPOS .

My problem is that when googling for it, if you insert the keywords "epos ride" then the first result is the design document... and this is absolutely not fine for my customers searching for the web app!

As long as I understand it, it is not possible to:

1. read_protect design documents (they contain fundamental information that must be accesible by the user, see. http://guide.couchdb.org/draft/design.html);
2. create a "redirect" from that page to the real index page (which only has a "/index.html" appendend to it), because the idea is exactly that through this RESTful API you get the design docs.

So I think the only way is to prevent google indexing design documents. Is that possible in some way? Are there any other solutions to solve this issue?

Answer 1

You can tell Google and also other search enginges, which parts of your web presence you don't want to be indexed.

Just put a file robots.txt in the root of your public reachable website, so all the robots can find it at http://www.example.com/robots.txt where www.example.com is your domain address.

This is a simple text file and below you find examples how to use the file to give commands to the web robots.

http://en.wikipedia.org/wiki/Robots_exclusion_standard says:

The standard specifies the instruction format to be used to inform the robot about which areas of the website should not be processed or scanned

This example tells all robots that they can visit all files because the wildcard * specifies all robots:

User-agent: *
Disallow:

This example tells all robots to stay away from one specific file:

User-agent: *
Disallow: /directory/file.html

This example tells all robots not to enter three directories:

User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/