SharePoint Crawl - Crawl Rules - Exclude

Question

We have lots of errors in Crawl Log about accessing one type of applications on URL.

I am getting an error message:

An unrecognized HTTP response was received when attempting to crawl this item. Verify whether the item can be accessed using your browser.

for URLs where the project ID is only changing:

• https://app.company.com/projects/01/TBase
• https://app.company.com/projects/02/TBase
• https://app.company.com/projects/03/TBase

and I want to create Crawl Rule, but haven't figured out the right formula for it.

Thank you for suggestions and ideas.

Answer 1

So you're looking for the RegEx to filter out URL's like the above where only the ID changes?

Try this, with the "use regular expression syntax for matching this rule" option checked.

^\w+://app.company.com/projects/\d{2}/TBase

If there are sometimes more than two digits you might use this instead.

^\w+://app.company.com/projects/\d+/TBase

Explanation: the important part is the \d+ or \d{2}, which matches with "one or more numbers" and "exactly 2 numbers", respectively.

Answer 2

Is app.company.com a sharepoint server or other? And, did you try the suggestion that was provided? Use run-as to run the browser as the same account as the account the crawler is using, and try to access the site. Do this from the SharePoint server to confirm there are no routing or firewall issues.