Question

In the robots.txt file, I am about to disallow some sections of my site.

For instance, I don't want my "terms and conditions" to be indexed by search engines.

 User-agent: *
 Disallow: /terms

The real path to the file is actually

 /data/terms_and_conditions.html

But I have used .htaccess to rewrite the URL.

Now to my Q, should I specify the rewritten url in the robots.txt or the actual url?

Follow-up question: Do I need to have an "allow" line too, or will the search engines assume all other is allowed which isn't in the robots.txt file?

Thanks

Was it helpful?

Solution

Search engines will assume that all other is allowed which isn't in the robots.txt. In your case it will disallow path /term.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top