Robots.txt block access to all https:// pages [closed]
-
16-09-2019 - |
Question
What would the syntax be to block all access to any bots to https:// pages? I have an old site that now doesn't have an SSL and I want to block access to all https:// pages
Solution
I don’t know if it works, if the robots use/request different robots.txt
for different protocols. But you could deliver a different robots.txt
for requests over HTTPS.
So when http://example.com/robots.txt
is requested, you deliver the normal robots.txt
. And when https://example.com/robots.txt
is requested, you deliver the robots.txt
that disallows everything.
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow