UrlMapping to static files in Grails

Question

I want to map the static files sitemap.xml and robots.txt which a located in my web-app directory. The urls should be as follows:

http://www.mydomain.com/sitemap.xml 
http://www.mydomain.com/robots.txt

How do I have to set the url mapping to make these routes work?

Answer 1

I use this mapping for robots.txt:

"/robots.txt" (view: "/robots")

And then have a grails-app/views/robots.gsp that contains the content for robots.txt. This way I can use <g:if env="..."> to easily have different content for different environments.

In order for this to work for a ".xml" extension, you need to change the Content Negotiation config.

grails.mime.file.extensions = false // disables the parsing of file extensions from URLs into the request format

Answer 2

The simplest way is to tell grails to ignore them in UrlMappings.groovy:

class UrlMappings {
    static excludes = ['/robots.txt', '/sitemap.xml']

    static mappings = {
        // normal mappings here ...
    }
}

Answer 3

It might also be helpful to to set up a nofollow to your Staging environment if you are using one. Not sure if there is a use case to having a staging site indexed.... so if you agree you may be able to use these steps to help block that.

If you are using Tomcat, set an environment variable such as NOFOLLOW=true --> see here for example: TOMCAT_OPTS, environment variable and System.getEnv()

Next as mentioned by @doelleri set the urlMappings

UrlMappings

//Robots.txt
"/robots.txt"(controller: 'robots', action:'robots')

Then use your robotsController to detect the environment variable you set on your staging tomcat.

RobotsController

def robots() {
    if (System.getenv('NOFOLLOW') == 'true') {
        def text = "User-agent: *\n" +
            "Disallow: /cgi-bin/ \n" +
            "Disallow: /tmp/ \n" +
            "Disallow: /junk/ \n" +
            "Disallow: /admin/ \n" +
            "Crawl-delay: 5 \n" +
            "Sitemap: https://www.example.com/sitemap.xml"

        render(text: text, contentType: "text/plain", encoding: "UTF-8")
    } else {
        render(status: 404, text: 'Failed to load robots.txt')
    }
}

robots.gsp

<%-- Content rendered from controller -> so leave blank :) --%>