Whats the correct way to use robots.txt

Question

I am trying to get robots.txt to work so that search engines start indexing my website and show meta info like descriptions etc.

However, I get this message:

A description for this result is not available because of this site's robots.txt – learn more.

Here is what my robots.txt look like.

# See http://www.robotstxt.org/wc/norobots.html for documentation on how to use the robots.txt file
#
# To ban all spiders from the entire site uncomment the next two lines:
User-agent: *
Disallow: /tags/*
Disallow: /users/*

What do I need to change?

This is a Rails4 application hosted on Heroku and is in the public directory in the Rails repository

Answer 1

First of all, it is not compulsory to use robots.txt file! you only need to use them in case you don't want the search engines to crawl specific pages or directories of your website.

In your case, you are restricting search engines to crawl tags and users' directories hosted on the root. Now, any page inside this directory will give this error.

I also recommend using the Google webmaster tool and verifying your website. You can test Robots.txt file from there.

Answer 2

Try removing some asterisks:

User-agent: *
Disallow: /tags/
Disallow: /users/   

Meanwhile, providing a location to your site map might be helpful too:

Sitemap: www.yoursite.com/sitemap.xml