Crawler authentication in SharePoint 2013 for STS based external website

Question

I am not able to crawl website that has STS (security token service) based authentication using FAST Search for SharePoint 2013 . I tried Form credentials and Cookie based authentication in Crawl Rules but following error comes:

Error: Access is denied. You may not have sufficient privileges to perform the operation.

What is the right way to crawl an external web that is secured with STS based authentication?

Answer 1

I explored web in order to solve this issue but it seems it is not possible to crawl an external website using SharePoint search that has STS based authentication and don't have Windows authentication enabled. Here are few references:

"If you are using claims-based authentication, make sure that that Windows authentication is enabled on any Web applications to be crawled." Source

"SharePoint can only crawl using a Windows NTLM authentication account. Your content source must authorize the Windows account sent as part of the crawl request in order to access the document content. Though claims authentication is supported in SharePoint 2010, the gatherer is still not a claims-aware application and will not access a content source that has claims authentication only." Source

"No, it needs NTLM/Kerberos in order to crawl the Web App." Source

Answer 2

You need to create a new crawler rule with default crawler account if it already has read permission on external websites, if not then you can specify a new crawler rule with a crawler account that has at least read permission on external content your crawler will crawl.

Since you getting Access denied error seems like your crawler account doesn't have permissions to crawl external websites regardless of authentication type.

Other then that,

Ensure that the domain account that is used for the default content access account or any other content access account is not the same domain account that is used by an application pool associated with any Web application that you crawl. Doing so can cause unpublished content in SharePoint sites and minor versions of files (that is, history) in SharePoint sites to be crawled and indexed.

Source

Hope it helps.

Answer 3

There is one more TechNet reference for SharePoint 2013 which supports the need for Windows NTLM for Search Crawling.

Jamil already quotes Plan crawling and federation in SharePoint Server 2013 from TechNet's SharePoint 2013 Authentication Planning.

Here's another reference from a different page in the same section:

• Plan for user authentication methods in SharePoint 2013, Microsoft TechNet, SharePoint 2013 Authentication Planning Guide

The crawl component requires NTLM to access content. At least one zone must be configured to use NTLM authentication. If NTLM authentication is not configured on the default zone, the crawl component can use a different zone that is configured to use NTLM authentication.