How do i configure nginx to redirect to a url for robots.txt & sitemap.xml
-
11-09-2019 - |
Question
I am running nginx 0.6.32 as a proxy front-end for couchdb. I have my robots.txt in the database, reachable as http://www.example.com/prod/_design/mydesign/robots.txt. I also have my sitemap.xml which is dynamically generated, on a similar url.
I have tried the following config:
server {
listen 80;
server_name example.com;
location / {
if ($request_method = DELETE) {
return 444;
}
if ($request_uri ~* "^/robots.txt") {
rewrite ^/robots.txt http://www.example.com/prod/_design/mydesign/robots.txt permanent;
}
proxy-pass http://localhost:5984;
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
}
This appears to work as a redirect but is there a simpler way?
Solution
Or you can put it simply in its own location -
location /robots.txt {
alias /Directory-containing-robots-file/robots.txt;
}
OTHER TIPS
I think you want to setup a location. The key part being the "=" directive.
location = /robots.txt {
alias /your_full_path/robots.txt ;
}
(When I didn't include the =, it still went to the default nginx root location)
Setting it like below, would cause all /robots.txt* requests to be read out of /var/foo. So /robots.txt.bing tries reading /var/foo/robots.txt.bing off of disk. The "^~" indicates that it is a regular expression match on the beginning of the request.
location ^~ /robots.txt {
root /var/foo;
}
rewrite with http://... is inteded to do the redirect. You are probably looking for:
rewrite ^/robots.txt /prod/_design/mydesign/robots.txt last;
This should work also, and may perform slightly better.
location /robots.txt { alias /var/www/django/mysite/static/robots.txt; }
Note that the first parameter is a regular expression so you should probably escape . and match end of line.
location / {
rewrite ^/robots\.txt$ /static/robots.txt last;
...
}
Considering that robots.txt
is located at /var/www/mysite/static/robots.txt
, the following worked for me:
location = /robots.txt {
root /var/www/mysite/static/;
}