Search engines not able to index asp.net site due to 302 redirects to Error page

 

I am having the hardest time determining the cause of our Asp.net site not getting indexed in search engines - the entire site. When I use google's "Fetch As Googlebot" tool, it throws the below error. I made sure my site works with sessions disabled (ie. setting sessionMode="Off" in web.config), I've googled all over the web, and still no luck.

Here's the error I am getting from googlebot: 

 HTTP/1.1 302 Found
 Date: Thu, 02 Dec 2010 23:05:49 GMT
 Server: Microsoft-IIS/6.0
 X-Powered-By: ASP.NET
 X-AspNet-Version: 2.0.50727
 Location: /ErrorPage.aspx?aspxerrorpath=/Default.aspx
 Cache-Control: private
 Content-Type: text/html; charset=utf-8
 Content-Length: 168

 <html><head><title>Object moved</title></head><body>
 <h2>Object moved to <a href="%2fErrorPage.aspx%3faspxerrorpath%3d%2fDefault.aspx">here</a>.</h2>
 </body></html>

I found the answer myself.

Make sure to check Request.UserLanguages != null before using it. Also, make sure the CurrentCulture is set to a valid default value. The reason for both these checks is because bots don't use Request.UserLanguages - it's always null. Browsers do use Request.UserLanguages. To restate it in other words: Don't set the CurrentCulture if Request.UserLanguages is null.

Here's the problem:

All the aspx pages in my site inherit from a custom base class that inherits from System.Web.UI.Page. This isn't a problem until you override the OnLoad() or init events with code that throws an exception for bots only. I had this line of code in my OnLoad() event:

Thread.CurrentThread.CurrentCulture = CultureInfo.CreateSpecificCulture(Request.UserLanguages[0]);

This code works great when accessing aspx pages from a browser. Duh! who isn't going to use a browser...? answer: googlebot and all other bots.

How to check if googlebot has a problem indexing your site:

  • Sign up for google webmastertools if you haven't already.
  • Go to Labs -> Fetch As Googlebot
  • Type in the url you want to check. Then click the status link to see what googlebot found. If googlebot returned any redirects (like 302's) that's usually not a good thing. If all the pages on your site are getting 302 redirects to your custom error page, then you have a problem. What it means is your web pages are throwing an error (an unhandled exception) whenever googlebot tries to access them. Look through your Page_Load() and Init() functions for errors googlebot might have. You can also test your pages using a cool command line app called curl (http://curl.haxx.se/). Using this tool I was able to test the site on our test server before releasing to production (which is what you'd have to do every time you make a change when using google fetch).
    • 链接地址: http://www.djcxy.com/p/49224.html

    上一篇: Javascript链接等待弹出窗口返回

    下一篇: 由于302重定向到错误页面，搜索引擎无法为asp.net网站编制索引