Search engines not able to index asp.net site due to 302 redirects to Error page

StackOverflow https://stackoverflow.com/questions/4369917

  •  09-10-2019
  •  | 
  •  

Question

I am having the hardest time determining the cause of our Asp.net site not getting indexed in search engines - the entire site. When I use google's "Fetch As Googlebot" tool, it throws the below error. I made sure my site works with sessions disabled (ie. setting sessionMode="Off" in web.config), I've googled all over the web, and still no luck.

Here's the error I am getting from googlebot:

 HTTP/1.1 302 Found
 Date: Thu, 02 Dec 2010 23:05:49 GMT
 Server: Microsoft-IIS/6.0
 X-Powered-By: ASP.NET
 X-AspNet-Version: 2.0.50727
 Location: /ErrorPage.aspx?aspxerrorpath=/Default.aspx
 Cache-Control: private
 Content-Type: text/html; charset=utf-8
 Content-Length: 168

 <html><head><title>Object moved</title></head><body>
 <h2>Object moved to <a href="%2fErrorPage.aspx%3faspxerrorpath%3d%2fDefault.aspx">here</a>.</h2>
 </body></html>
Was it helpful?

Solution

I found the answer myself.

Make sure to check Request.UserLanguages != null before using it. Also, make sure the CurrentCulture is set to a valid default value. The reason for both these checks is because bots don't use Request.UserLanguages - it's always null. Browsers do use Request.UserLanguages. To restate it in other words: Don't set the CurrentCulture if Request.UserLanguages is null.

Here's the problem:

All the aspx pages in my site inherit from a custom base class that inherits from System.Web.UI.Page. This isn't a problem until you override the OnLoad() or init events with code that throws an exception for bots only. I had this line of code in my OnLoad() event:

Thread.CurrentThread.CurrentCulture = CultureInfo.CreateSpecificCulture(Request.UserLanguages[0]);

This code works great when accessing aspx pages from a browser. Duh! who isn't going to use a browser...? answer: googlebot and all other bots.

How to check if googlebot has a problem indexing your site:

  1. Sign up for google webmastertools if you haven't already.
  2. Go to Labs -> Fetch As Googlebot
  3. Type in the url you want to check. Then click the status link to see what googlebot found. If googlebot returned any redirects (like 302's) that's usually not a good thing. If all the pages on your site are getting 302 redirects to your custom error page, then you have a problem. What it means is your web pages are throwing an error (an unhandled exception) whenever googlebot tries to access them. Look through your Page_Load() and Init() functions for errors googlebot might have. You can also test your pages using a cool command line app called curl (http://curl.haxx.se/). Using this tool I was able to test the site on our test server before releasing to production (which is what you'd have to do every time you make a change when using google fetch).
Licensed under: CC-BY-SA with attribution
Not affiliated with StackOverflow
scroll top