I don't want to see Robots.txt in result list of search. I'm working on an asp.net project.

I found some failed solutions, one of them like below:

  • Using "X-Robots-Tag" in header.

But I want to hide a txt file so there is no "header" part in it.

Thanks for help.

有帮助吗?

解决方案

The capitalization is important to most popular search engines, including Google (source). Change your file name from Robots.txt to robots.txt. If you name it correctly, most popular search engines (Bing, Yahoo, Google, etc.) will not index it. No extra effort on your part.

Here is Yahoo's crawler help that describes case-sensitivity.

Here is the standard that describes common courtesy for crawlers with regards to robots.txt. It is explicitly stated that empty robots.txt files are ignored completely, and it is common sense that they are not indexed. Otherwise, we would see billions of robots.txt search results in Google.

其他提示

A txt file sent via http has a header(containing type:text/plain) just take request and send txt with your own headers.

许可以下: CC-BY-SA归因
不隶属于 StackOverflow
scroll top