Todd Klindt's Office 365 Admin Blog

Quick Launch

All Site Content

Todd Klindt's home page > Todd Klindt's Office 365 Admin Blog > Posts > Can’t crawl web apps you KNOW you should be able to crawl

February 17

Can’t crawl web apps you KNOW you should be able to crawl

by Todd O. Klindt

on 2/17/2009 11:56 AM

Category: SharePoint

This has come up so many times in the last couple of days I just HAVE to blog about it. I was originally going to call this post "the best registry setting ever" but I realize that would make it tough for search engines to point people here when they need it. Here's the general problem description, you've got a SharePoint environment and you're getting an "access denied" trying to access something you KNOW you should have access to. In the cases I've seen this has been trying to crawl a web app, or trying to create a My Site. In both cases the App Pool ID has the correct permissions, and in the case of search the default content account has been given "Read only" permission to the web app via a web app policy. I was getting this error in the context of the My Site problem. No one could create a My Site. A user couldn't. A Farm Admin couldn't. A Domain Administrator couldn't. This really kicked my butt for a couple of days. Yesterday a friend of mine contacted me. She had the search issue. I walked her through some stuff and she had it all configured correctly. Despite all that, when she ran a crawl, she got a permissions error. I bounced this off of my side-kick, and search aficionado Shane Young. He asked if I'd tried the "loopback fix." Before yesterday to me loopback was only a network thing. It was a 127 address, or it was a phony network adapter I added. He pointed me to my favorite KB ever (or at least since yesterday) and told me to try Method #1. Without further ado, here is the greatest KB article ever.

You receive error 401.1 when you browse a Web site that uses Integrated Authentication and is hosted on IIS 5.1 or IIS 6

Let's all take a moment to absorb the greatness.

Okay, now that we've got that out of our systems let's dig into this a little. While the KB references Windows 2003 all four of the issues I've fixed with it have been on Windows 2008, so don't let the "applies to" fool you. Both 2003 and 2008 have a security measure that disallows loopback communications in case your machine has any errant processes on it that are trying to attack it. That's good from a security standpoint, but it can break SharePoint if you have multiple things running on the same box. Using Method 1 in that KB and disabling the Loopback Check restores order to your SharePoint environment. Since I learned about it yesterday it has fixed problems in four different environments. Go KB 896861!

If you're getting weird permissions errors on your SharePoint farm this is worth trying. It's easily reversible if it doesn't fix your issue. If you have security folks in your environment you should probably run it past them too, just in case. J

14 Comment(s)

Comments

Been there done that :)

I had the same issue on a couple of farms. It seems to be an issue with .NET Framework 3.5 SP1, which became available on Windows update some weeks ago.

Before this update, there was no problems with crawling on the sites.

Disabling loopback, fixes everything magically :-)

on 2/18/2009 1:56 AM

Good read!

If your farm has a separate index server, it seems an increasingly popular practice to put the web front end role on that index and have the index server crawling its own web front end (using a hosts file entry rather than using the "use specific server for crawling" configuration"). When this is the farm architecture and configuration, I've found it a good troubleshooting technique for this problem to change the hosts file entry so that the indexer crawls a different web front end server in the farm. If it then crawls fine, you know it is likely the DisableLoopbackCheck that is causing the problem.

It's important to note that the need for DisableLoopbackCheck has been around in IIS for years. I can't recall if it was IIS 4 or IIS 5 that first exhibited the problem. Interestingly, there has apparently been a recent change to the way that the loopback check is implemented. When the problem first appeared, a web server administrator logged in through Terminal Services could simply open a browser and see the 401 error in the browser and know that this registry change was necessary. However, with the current implementation of the loopback check, an interactive logon session does not seem to be subject to the loopback check, only non-interactive sessions. Therefore the administrator can no longer simply log into the server and verify the error in the browser.

FWIW, it's also important to keep in mind that by making this change, the security of the server has been loosened although to what extent will certainly be debated. There was obviously a security reason for the IIS developers to put it in, so taking it out has implications. For this reason, it is helpful to avoid using this registry change, if you can, on a server which faces the users. In our case, the index server does not face the internet and is not in the load balancer.

I hope this information is helpful. If not, Todd, please feel free to delete.

Mark Kordelski
http://twitter.com/delsk

on 2/23/2009 11:37 AM

PowerShell Script to do the Registry Edits

I made a powershell script that does the two registry edits needed to turn off Loopback checking.
Check it out:
http://blumenthalit.net/blog/Lists/Posts/Post.aspx?ID=131

on 7/11/2009 11:49 PM

Re: PowerShell Script to do the Registry Edits

Thanks Michael.
As a reminder, before making this change in production make sure you understand the security issue you may be exposing yourself to.

tk

Todd O. Klindt

on 7/12/2009 1:51 PM

Best blog post ever

After two days of frustration, you saved me. Thanks!

Jim Lehmer
http://twitter.com/dullroar

on 9/23/2009 10:47 AM

Re: Can’t crawl web apps you KNOW you should be able to crawl

i have 2 pdf documents which were uploaded at the same time almost 2 years ago and the properties of the 2 documents match in every way. one is found in a search and the other is not ...... very confused. any ideas ?

on 2/11/2010 5:38 AM

Re: Re: Can’t crawl web apps you KNOW you should be able to crawl

Could be a bunch of things. What does it say about the document that doesn't crawl in your crawl log?

Todd O. Klindt

on 2/11/2010 10:43 AM

Thank you very much

Thanks alot for your help.

on 5/5/2010 9:00 PM

Re: Thank you very much

Glad to help.

tk

Todd O. Klindt

on 5/5/2010 9:10 PM

Problem on Server 2008

In the http://support.microsoft.com/kb/281308/ it only mentions Server 2000 and 2003 servers. Does this also apply to Server 2008?

Thanks for your input

Jerrod.

on 8/19/2010 1:08 PM

1 - 10