How to let or restrict Google Bot index or crawl certain things in Wordpress?

Question

Well, I have problem with Google Bot. Taking 700MB of bandwidth daily. This is for those which will obviously ask why I want to do this.

I know about robots.txt and that I can stop bots to index some folders. But what in WordPress, I am using post-name permalinks, so permalinks for posts and pages are just /page or /post.

Searched for any plugin to restrict bot on indexing only few tags and few categories, didn't found it.

Want to allow sticky posts, few categories, few tags. Can be done? How?

I have update on this question. I decided to go with robots.txt rulles.

User-agent: *
Disallow: /

User-agent: AhrefsBot
Disallow: /

User-agent: AhrefsBot/3.1
Disallow: /

User-agent: Yahoo-slurp
Disallow: /

User-agent: Msnbot
Disallow: /

User-agent: Googlebot
Allow: /
Disallow: /category
Disallow: /video
Disallow: /author
Disallow: /?s=
Disallow: /feed/
Disallow: /xmlrpc.php
Disallow: /cgi-bin/
Disallow: /wp-admin/
Disallow: /wp-includes/
Disallow: /wp-content/plugins/
Disallow: /wp-content/cache/
Disallow: /wp-content/themes/
Disallow: /wp-login.php
Disallow: /wp-register.php
Disallow: /tag
Allow: /tag/marry
Allow: /tag/john

Will last two tags be indexed? And is there something more to hide in WordPress?

Answer 1

If you want to allow particular posts but disallow everything else, then use Allow tags. For example:

User-agent: Googlebot
Allow: /post/foo
Allow: /page/bar
Disallow: *

So the bot can crawl the pages you specify, but not anything else.