robots.txt content itself is indexed? [closed]

Question

Closed. This question is off-topic. It is not currently accepting answers.

---

Want to improve this question? Update the question so it's on-topic for Stack Overflow.

Closed 8 years ago.

Improve this question

The contents of my robots.txt file are actually itself indexed and show up in Google search results. It's only Google and not Yahoo for example.

I really think Google should understand not to index the contents of my robots file as it's only there to tell Google what not to index!

Am I missing something here?

Answer 1

That's normal. Mine was indexed a few months ago too. I think it's an issue with the crawler. edit: btw, you can delete the link on the webmaster tools website.

Answer 2

I wouldn't recommend disallowing the robots.txt file from itself. Instead you can use the X-Robots-Tag http header with a value of noindex which will prevent the search engines from Indexing the file.

This will allow them access to your robots.txt but prevent it from showing up in the search results.

Answer 3

You can add Disallow: /robots.txt to your robots.txt. I think it should avoid the robots.txt itself from being indexed.

Answer 4

I Think that disallowing your own robots.txt file (if possible) will result in all the other commands written there to be ignored by bots also.