ACCESS DENIED & CRAWL ERROR IN INDEX SERVER/SEARCH SERVER

This is a quick blog because I encountered this problem and had it solved quickly enough. Recently took down my index server because NT found it wasn't up to spec (not sharepoint spec but NTs spec such as age of the server, it should be getting ready to enter into the archived servers). So, had to remove the index and when a fresh server came on board last week, promptly configured it and ran into this problem. So, here it is:
Windows Server 2003 SP1 introduced a loopback security check. This feature is obviously also present in Windows Server 2008. The feature prevents access to a web application using a fully qualified domain name (FQDN) if an attempt to access it takes place from a machine that hosts that application. The end result is a 401.1 Access Denied from the web server and a logon failure in the event log.

The trouble is there are also scenarios where this fix will break normal operations of SharePoint.
1. Search Indexing.
If you are hosting the WSS Web Application Service on your Indexer for the purposes of having a “Dedicated Crawl Front End” and avoiding a network hop. This is common in small scale “Medium Server Farms”. Because the Indexer is crawling itself, the crawl log will fill up with 401s and your content won’t get indexed.

Crawl Access Denied Error

2. Web Application “Warm Ups”.
If you are running a scheduled task or timer job to hit the Web Application to avoid the start up lag after an application pool recycle, the “warm up” will fail with a 401.

3. Custom Code using SharePoint Web Services.
If you have custom code, either in SharePoint or out with it that leverages SharePoint Web Services (such as using the ExcelService API) these requests will fail with a 401.

Workaround

If you are working on a development environment or on just a single MOSS box – go for it – disable it completely. You need to debug and test locally and it’s likely you don’t know what addresses you will use ahead of time. I as a matter of course disable the check as part of my sysprep build for all my development and test machines. I never hit the problem because my base image is all sorted as I want it. I recommend you do the same.

1. Login to the SharePoint server.
2. Click Start, click Run, type regedit, and then click OK.
3. In Registry Editor, locate and then click the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa
4. Right-click Lsa, point to New, and then click DWORD Value.
5. Type DisableLoopbackCheck, and then press ENTER.
6. Right-click DisableLoopbackCheck, and then click Modify.
7. In the Value data box, type 1, and then click OK.
8. Quit Registry Editor, and then restart your computer.

However, for production environments, DO NOT DISABLE this feature. You are unpicking a serious security check of the OS. If that environment underwent a security audit by a competent security engineer, it would be flagged. You should add a list of addresses you wish to exclude. This makes your scenario work whilst retaining the security check.

1. Login to the SharePoint server.
2. Click Start, click Run, type regedit, and then click OK.
3. In Registry Editor, locate and then click the following registry key: HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\Lsa\MSV1_0
4. Right-click MSV1_0, point to New, and then click Multi-String Value.
5. Type BackConnectionHostNames, and then press ENTER.
6. Right-click BackConnectionHostNames, and then click Modify.
7. In the Value data box, type the host name or the host names for the sites that are on the local computer, and then click OK.
8. Quit Registry Editor, and then restart the IISAdmin service.

References
Microsoft KB 896861
harbar.net
mossgurus.com
nishantrana.wordpress.com
social.technet.microsoft.com article I
social.technet.microsoft.com article II