Implementing Sitecore Multisite Robots.txt files

Question

How to implement to have different robots.txt files for each website hosting on the same Sitecore solution. I want to read dinamically robots.txt from sitecore items.

Answer 1

you need to follow next steps:

1) Create and implement your custom generic (.ashx) handler.

2) In the web.config file add the following line to the section

3) Navigate to the section and add here

4) On home item you will have "Robots" field (memo, or multi line field, not richText field) Your custom generic handler will look like :

 public class Robots : IHttpHandler
{

    public virtual void ProcessRequest(HttpContext context)
    {
        private string defaultRobots = "your default robots.txt content ";

        string robotsTxt = defaultRobots;

        if ((Sitecore.Context.Site == null) || (Sitecore.Context.Database == null))
        {
            robotsTxt = defaultRobots;
        }
        Item itmHomeNode = Sitecore.Context.Database.GetItem(Sitecore.Context.Site.StartPath);
        if (itmHomeNode != null)
        {
            if ((itmHomeNode.Fields["Robots"] != null) && (itmHomeNode.Fields["Robots"].Value != ""))
            {
                robotsTxt = itmHomeNode.Fields["Robots"].Value;
            }
        }

        context.Response.ContentType = "text/plain";
        context.Response.Write(robotsTxt);

    }

Answer 2

We had similar problems especially in the multi site environment, so we used the handlers for implementing robots.txt

Create a new class inheriting from IHTTPHandler and implement the logic within the process method. Write the XML ouput to the context object.

context.Response.ContentType = "text/plain";
context.Response.Output.Write({XML DATA});

Add the custom handler and trigger.

  <handler trigger="~/Handlers/" handler="robots.txt"/>

  <add name="{Name}" path="robots.txt" verb="*" type="{Assembly Name and Type}" />

Answer 3

It seems that if you want to access Sitecore Context, and any items, you need to wait untill this stuff is resolved. The aboce method will always give you a null in the Site definition, as this isnt resolved when the filehandler kicks in.

It seems that to get the Sitecore.Context, you should implement a HttpRequestProcessor in Sitecore, that renderes the robots.txt, example on this website: http://darjimaulik.wordpress.com/2013/03/06/how-to-create-handler-in-sitecore/

Answer 4

You can refer to this blog post for step-by-step explanation on how to do it with a custom HttpRequestProcessor and a custom robots settings template : http://nsgocev.wordpress.com/2014/07/30/handling-sitecore-multi-site-instance-robots-txt/