prevent google from indexing

Question

hi sirs what's the best way to prevent google from showing of a folder in the search engine ?, like e.g www.example.com/support , what should i do if I want the support folder to disappear in google ? the first thing I did was place a 'robots.txt' file and include this code

User-agent: *
Disallow: /support/etc

but the results is a total disaster, am not able to use the support page anymore unless i remove the robots.txt what's the best thing to do ?

Answer 1

robots.txt shouldnt affect the way your page function. If in doubt, you can use tools to generate like http://www.searchenginepromotionhelp.com/m/robots-text-creator/simple-robots-creator.php or http://www.seochat.com/seo-tools/robots-generator/

When dissallowing in robots file, you can explicitly specify a file or subfolder rather than just a folder.

You can also use meta tag in your document to tell the crawler not to use it

<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">

Answer 2

what's the best way to prevent google from showing of a folder in the search engine ?

A robots.txt file is the right way to do this. Your example is correct for blocking the /support/etc directory and its descendants.

am not able to use the support page anymore unless i remove the robots.txt

It doesn't make sense that a robots.txt file would affect the way your site functions, and certainly it should never affect which pages can be accessed by a human. I suspect something else is awry -- check your server logs to see what kinds of errors are being recorded.

Answer 3

While not the preferred method of limiting robot access, Google talks about using a noindex meta tag here. This will also prevent the various pages from showing up if they are linked to by a site other than your own.

A good discussion of limiting bots that visit your site can be found here.