Question

I am seeing thousands of warnings in my crawl log for a login page for a public website that is using a complex URL.

Is there any benefit then to create an exclusion crawl rule for this login page specifically?

Will search crawl faster or not take a long if I use the exclusion crawl rule for the login page?

Was it helpful?

Solution

By default, the crawler will not follow complex URLs, which are URLs that contain a question mark followed by additional parameters — for example, http://contoso/page.aspx?x=y. If you enable the crawler to follow complex URLs, this can cause the crawler to gather many more URLs than is expected or appropriate. This can cause the crawler to gather unnecessary links, fill the crawl database with redundant links, and result in an index that is unnecessarily large.

Performance will improve if you create exclusion rules. And it is one of the best practice

https://technet.microsoft.com/en-us/library/dn535606.aspx#BKMK_UseCrawlRules

OTHER TIPS

Exclusion rules are useful and help in performance.

Avoid crawling less relevant content by excluding one or more URLs. This also helps to reduce the use of server resources and network traffic, and to increase the relevance of search results.Creating crawl rules to exclude URLs is particularly useful when planning start addresses for external content because the impact on resource usage is not under the control of people in your organization.

https://technet.microsoft.com/en-us/library/cc262531%28v=office.12%29.aspx?f=255&MSPPError=-2147217396

Licensed under: CC-BY-SA with attribution
Not affiliated with sharepoint.stackexchange
scroll top