Magento robots.txt 404s

Question

I have a Magento multi-site installation. I want to disallow just one of my stores, but not the others. I am trying to take advantage of the "Default Robots" option on the Magento config, so I left the default robots to "Index,Follow" for all stores except the one I want to disallow, which I changed to "Noindex,Nofollow". But when I go to my-URL/robots.txt, I get a 404 error.

Any idea why this might 404?

Can someone point my in the direction of how I might make a robots.txt that only disallows that one site?

Answer 1

This setting will only add a meta robots tag to your pages, e.g.

<meta name="robots" content="INDEX,FOLLOW" />

See: http://www.robotstxt.org/meta.html

The Magento application does not interface with your robots.txt