Question

is there any point in having a robots.txt with just one line of content like below

User-agent: *

compared to not having it at all ?

Was it helpful?

Solution

No, google (and let's not beat around the bush, Google is the search engine you should care about) by default will crawl everything unless you tell it not to.

If you want to help Google, use google webmaster tools and submit a sitemap in the correct format

http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156184

OTHER TIPS

No; there is no point in doing that.

Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top