How to hide Robots.txt in search engine results? [closed]

Question

Closed. This question needs details or clarity. It is not currently accepting answers.

---

Want to improve this question? Add details and clarify the problem by editing this post.

Closed 8 years ago.

Improve this question

I don't want to see Robots.txt in result list of search. I'm working on an asp.net project.

I found some failed solutions, one of them like below:

• Using "X-Robots-Tag" in header.

But I want to hide a txt file so there is no "header" part in it.

Thanks for help.

Answer 1

The capitalization is important to most popular search engines, including Google (source). Change your file name from Robots.txt to robots.txt. If you name it correctly, most popular search engines (Bing, Yahoo, Google, etc.) will not index it. No extra effort on your part.

Here is Yahoo's crawler help that describes case-sensitivity.

Here is the standard that describes common courtesy for crawlers with regards to robots.txt. It is explicitly stated that empty robots.txt files are ignored completely, and it is common sense that they are not indexed. Otherwise, we would see billions of robots.txt search results in Google.

Answer 2

A txt file sent via http has a header(containing type:text/plain) just take request and send txt with your own headers.