Pergunta

EDIT: Doing this with robots.txt isn't good enough for me.

This code block entire site from google. I do not know how to keep this code functional but allow google to access one folder on the site.

RewriteEngine On
RewriteBase /
RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteRule ^(.*)$ - [F,L]

Does this works if i had 2 conditions?

Options +FollowSymLinks -MultiViews
# Turn mod_rewrite on
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} Googlebot [OR]
RewriteCond %{HTTP_USER_AGENT} yahoo [OR]
RewriteCond %{REQUEST_URI} !^/allowed-folder/ [NC]
RewriteRule ^ - [F,L]
Foi útil?

Solução

As suggested in comments create a file robots.txt under DOCUMENT_ROOT with following content:

User-Agent: googlebot
Disallow: /
Allow: /allowed-folder/

If you really want to handle this via mod_rewrite then enable mod_rewrite and .htaccess through httpd.conf and then put this code in your .htaccess under DOCUMENT_ROOT directory:

Options +FollowSymLinks -MultiViews
# Turn mod_rewrite on
RewriteEngine On
RewriteBase /

RewriteCond %{HTTP_USER_AGENT} Googlebot [NC]
RewriteCond %{REQUEST_URI} !^/allowed-folder/ [NC]
RewriteRule ^ - [F,L]

Outras dicas

create file with name robots.txt

insert

User-agent: *
Disallow: /folder you dont wont google inside in/
Disallow: /other folder you dont wont google inside in/
Licenciado em: CC-BY-SA com atribuição
Não afiliado a StackOverflow
scroll top