FAST Search for SharePoint 2010: Batches Timing Out

Question

My organization has been attempting to use FAST Search for Sharepoint 2010 as a search solution for our DMS. I am currently utilizing a .Net indexing connector to index a database of ~ 8 million records and their accompanying files.

Previously, I was having a memory issue on the crawl box, which included the incremental crawls being unable to complete. After months of troubleshooting, that issue seemed to be resolved with a hotfix (http://support.microsoft.com/kb/2601211). The entire issue is illustrated in another thread (http://goo.gl/3CuZUY).

However, on the following incremental, I began to receive new warnings regarding batch time outs in the logs and an atrocious crawl rate (0.1 dps). This has also caused the crawls to be unable to complete, even after running for hundreds of hours. Yet again, I am at a loss.

I have been working with a Microsoft support rep for over two months, but they have been unable to offer assistance for any of our issues. I have also been monitoring the performance counters on both servers for quite some time, and (other than a high 'batches ready' number on the crawl box) I have yet to find a smoking gun. Thus, any suggestion would be greatly appreciated.

Environmental and log information detailed below:

SharePoint Environment

• 1x WFE
• 1x Database Server
• 1x Application Server (32GB RAM)
• 1x Index Server (FAST) (16GB RAM, 8 CPU Cores, 5 Document Processors)

Logs

======[FAST BOX]======

DOCLOG

• WARNING Document conversion failed: External process timeout reached (300 seconds) (warning code 0)
• INFO Processor "IFilterConverter" ran for 300s

PSCtrl Statistics

• IFilterConverter (System Time)229.6 (User Time)2670.5 (Real Time)778625.3

======[CRAWL BOX]======

ULS

• Timeout while feeding batch with 1 docs after 60.0s. Attempted 2 times: Could not submit content:operation set timed out
  [documentsubmitterworkerthread.cpp:492] d:\office\source\search\native\gather\plugins\contentpi\documentsubmitterworkerthread.cpp
• Content Exception after 151.8s: Could not submit content:WinHttpReceiveResponse failed. Url:'http:// [FAST SERVER]:13391/processing::session/5.2/1406916153000000043/process' Error:'12002'
• Timed out batch. Waiting 300 seconds before retrying [documentsubmitterworkerthread.cpp:521] d:\office\source\search\native\gather\plugins\contentpi\documentsubmitterworkerthread.cpp

Answer 1

We had a similar issue with FAST and we opened a support case as well. We were indexing 7 TB worth of data totaling around 3 million items. They said that disk IO was our major issue causing the timing out so we had to order new FAST servers. We ended up going with:

• 96 GB of RAM
• 2 x 6 Core Xeon CPUs
• 8 x 300 GB HD - 2 mirrored for OS, and other 6 in a RAID 50 for a little over 1 TB of space for the index

We got two of those to build our FAST farm and once we put them in place we haven't had any timeout issues with FAST.

With the new servers we increased our document processors to 24 and when we did our full indexing over the weekends it would take a little over 24 hours to complete.

Unfortunately, with FAST, it seems like the best answer is to throw more hardware at it.