How to stop all search engines, bots to crawl some urls

Question

I want to count ads click on a widget.

I've used on robot.txt file:

User-agent: *
Allow: /
Disallow: */ads_count/*

I've also add nofollow for all links in that widget.

But many bots still follow urls in that widget. I've get client ip to count urls and i got many IP form bots.

Answer 1

Did u try removing the (*) before */ads_count? As google documentation for SEO says, if you want to block all the bots, it's like u did:

User-agent: * // (to whom? (*) means all bots!
Disallow: /ads_count

Note that directives are case-sensitive. For instance, Disallow: /junk_file.asp would block http://www.example.com/junk_file.asp, but would allow http://www.example.com/Junk_file.asp. Googlebot will ignore white-space (in particular empty lines)and unknown directives in the robots.txt.

Answer 2

Allow and the wildcard * in Disallow are not part of the original robots.txt specification, so not all robots.txt parsers will know/note those rules.

If you want to block all pages starting with /ads_count/, you just need:

User-agent: *
Disallow: /ads_count/

However: not all bots respect robots.txt. So you'd still get hits by bad bots that ignore robots.txt.