robots.txt . To disallow everything in my test subdirectory

Question

I have a main website at mysite.com . I have created a duplicate of mysite.com and installed wordpress on : "mysite.com/test"

But recently , I just discovered that google indexes my test sit as well. and my sitemap contains link to those of the test site.

My robots.txt :

User-agent: *
Disallow: /site/wp-admin/
Disallow: /site/wp-includes/

Sitemap: https://mysite.com/sitemap.xml.gz

How do I go about making sure robots dont crawl mysite.com/test and everything in it? I also have to change my sitemap to only have links on my main site right?

Answer 1

How do I go about making sure robots dont crawl mysite.com/test and everything in it?

You put Disallow: /test in your robots.txt. (And don't include it in the sitemap, of course.)