robots.txt block bots crawling subdirectory [closed]

Question

Closed. This question does not meet Stack Overflow guidelines. It is not currently accepting answers.

---

This question does not appear to be about programming within the scope defined in the help center.

Closed 8 years ago.

Improve this question

I'd like to block all bots from crawling a sub directory http://www.mysite.com/admin plus any files and folders in that directory. For example there may be further directories inside /admin such as http://www.mysite.com/admin/assets/img

I'm not sure what is the exact correct declarations to include in robots.txt to do this.

Should it be:

User-agent: *
Disallow: /admin/

Or:

User-agent: *
Disallow: /admin/*

Or:

User-agent: *
Disallow: /admin/
Disallow: /admin/*

Answer 1

Based on information available on the net (I can't retrieve it all but some forums actually report the problem, as in here and here, for example) I'd follow those who suggest we never tell people or bots (or both) where is that we don't want them to look ("admin" looks like sensitive content...).

After having checked, I'd confirm it's the first one you say. Reference here