SharePoint 2013 Search “failed to acquire worker. Sandbox worker pool is closed”

Question

While attempting a new full crawl of a content source in SharePoint 2013 Search Administration, the Crawl Log URL View states the following:

The object was not found. ( Error parsing document '*******'. It was not possible to acquire a worker. Proxy '6' failed to acquire worker. Sandbox worker pool is closed.; ; SearchID = ******** )

The incremental crawls appear to be giving this error also. Rebooting the Search servers clears this issue one time.

I have followed the instructions in this post regarding the GPO and account permissions. https://social.technet.microsoft.com/Forums/sharepoint/en-US/ee997834-c508-48ca-8bac-25c3b5a86db1/crawl-errors-sandbox-worker-pool-is-closed?forum=sharepointadmin Specifically giving the Service account the following specific local user rights assignments to function fully, including:

• Adjust memory quotas for a process
• Impersonate a client after authentication
• Replace a process level token

Would like to try additional items before "removing and rebuilding the search service application from scratch".

Answer 1

Clearing the SharePoint Configuration Cache resolved this issue.

Per this article: http://blogs.msdn.com/b/josrod/archive/2007/12/12/clear-the-sharepoint-configuration-cache-for-timer-job-and-psconfig-errors.aspx

"If you experience issues with WSS and MOSS timer jobs failing to complete are receiving errors trying to run psconfig, clearing the configuration cache on the farm is a possible method for resolving the issue. The config cache is where we cache configuration information (stored in the config database) on each server in the farm. Caching the data on each server prevents us from having to make SQL calls to pull this information from the configuration database. Sometime this data can become corrupted and needs to be cleared out and rebuilt. If you only see a single server having issues, only clear the config cache on that server, you do not need to clear the cache on the entire farm. To do a single server, follow the steps below on just the problem server.

To clear the config cache on the farm, follow these steps:

1. Stop the OWSTIMER service on ALL of the MOSS servers in the farm.
2. On the Index server, navigate to: Server 2003 location: Drive:\Documents and Settings\All Users\Application Data\Microsoft\SharePoint\Config\GUID and delete all the XML files from the directory.
   Server 2008 location: Drive:\ProgramData\Microsoft\SharePoint\Config\GUID and delete all the XML files from the directory.
3. Delete all the XML file in the directory. NOTE: ONLY THE XML FILES, NOT THE .INI FILE.
4. Open the cache.ini with Notepad and reset the number to 1. Save and close the file.
5. Start the OWSTIMER service on the Index server and wait for XML files to begin to reappear in the directory.
6. After you see XML files appearing on the Index server, repeat steps 2, 3 & 4 on each query server, waiting for XML files to appear before moving to subsequent servers.
7. After all of the query servers have all been cleared and new .xml files have been generated, proceed to the WFE and Application servers in the farm, following steps 2, 3, 4 and 5 for each remaining server."

Here is a PowerShell script to automate the process. It is recommended to clear the cache every two weeks.

# Clear the SharePoint Timer Cache
#
# 2009 Mickey Kamp Parbst Jervin (mickeyjervin.wordpress.com)
# 2011 Adapted by Nick Hobbs (nickhobbs.wordpress.com) to work with SharePoint 2010,
# display more progress information, restart all timer services in the farm,
# and make reusable functions.
# 2012 Adapted by Dwayne Selsig (www.dwayneselsig.eu) to also work with SharePoint 2013. 
# Added loading of SharePoint Snapin for Powershell. Also this script only queries 
# the SharePoint servers.

# Output program information
Write-Host -foregroundcolor White ""
Write-Host -foregroundcolor White "Clear SharePoint Timer Cache"

#**************************************************************************************
# Constants
#**************************************************************************************
Set-Variable timerServiceName -option Constant -value "SPTimerV4"
Set-Variable timerServiceInstanceName -option Constant -value "Microsoft SharePoint Foundation Timer"

#**************************************************************************************
# Functions
#**************************************************************************************
#<summary>
# Loads the SharePoint Powershell Snapin.
#</summary>
Function Load-SharePoint-Powershell
{
    If ((Get-PsSnapin |?{$_.Name -eq "Microsoft.SharePoint.PowerShell"})-eq $null)
    {
            Write-Host -ForegroundColor White " - Loading SharePoint Powershell Snapin"
        Add-PsSnapin Microsoft.SharePoint.PowerShell -ErrorAction Stop
    }
}

#<summary>
# Stops the SharePoint Timer Service on each server in the SharePoint Farm.
#</summary>
#<param name="$farm">The SharePoint farm object.</param>
function StopSharePointTimerServicesInFarm($farm)
{
Write-Host ""

# Iterate through each server in the farm, and each service in each server
foreach($server in $farm)
{
foreach($instance in $server.ServiceInstances)
{
# If the server has the timer service then stop the service
if($instance.TypeName -eq $timerServiceInstanceName)
{
[string]$serverName = $server.Name

Write-Host -foregroundcolor DarkGray -NoNewline "Stop '$timerServiceName' service on server: "
Write-Host -foregroundcolor Gray $serverName

$service = Get-WmiObject -ComputerName $serverName Win32_Service -Filter "Name='$timerServiceName'"
sc.exe \\$serverName stop $timerServiceName > $null

# Wait until this service has actually stopped
WaitForServiceState $serverName $timerServiceName "Stopped"

break;
}
}
}

Write-Host ""
}

#<summary>
# Waits for the service on the server to reach the required service state.
# This can be used to wait for the "SharePoint 2010 Timer" service to stop or to start
#</summary>
#<param name="$serverName">The name of the server with the service to monitor.</param>
#<param name="$serviceName">The name of the service to monitor.</param>
#<param name="$serviceState">The service state to wait for, e.g. Stopped, or Running.</param>
function WaitForServiceState([string]$serverName, [string]$serviceName, [string]$serviceState)
{
Write-Host -foregroundcolor DarkGray -NoNewLine "Waiting for service '$serviceName' to change state to $serviceState on server $serverName"

do
{
Start-Sleep 1
Write-Host -foregroundcolor DarkGray -NoNewLine "."
$service = Get-WmiObject -ComputerName $serverName Win32_Service -Filter "Name='$serviceName'"
}
while ($service.State -ne $serviceState)

Write-Host -foregroundcolor DarkGray -NoNewLine " Service is "
Write-Host -foregroundcolor Gray $serviceState
}

#<summary>
# Starts the SharePoint Timer Service on each server in the SharePoint Farm.
#</summary>
#<param name="$farm">The SharePoint farm object.</param>
function StartSharePointTimerServicesInFarm($farm)
{
Write-Host ""

# Iterate through each server in the farm, and each service in each server
foreach($server in $farm)
{
foreach($instance in $server.ServiceInstances)
{
# If the server has the timer service then start the service
if($instance.TypeName -eq $timerServiceInstanceName)
{
[string]$serverName = $server.Name

Write-Host -foregroundcolor DarkGray -NoNewline "Start '$timerServiceName' service on server: "
Write-Host -foregroundcolor Gray $serverName

$service = Get-WmiObject -ComputerName $serverName Win32_Service -Filter "Name='$timerServiceName'"
sc.exe \\$serverName start $timerServiceName > $null

WaitForServiceState $serverName $timerServiceName "Running"

break;
}
}
}

Write-Host ""
}

#<summary>
# Removes all xml files recursive on an UNC path
#</summary>
#<param name="$farm">The SharePoint farm object.</param>
function DeleteXmlFilesFromConfigCache($farm)
{
Write-Host ""
Write-Host -foregroundcolor DarkGray "Delete xml files"

[string] $path = ""

# Iterate through each server in the farm, and each service in each server
foreach($server in $farm)
{
foreach($instance in $server.ServiceInstances)
{
# If the server has the timer service delete the XML files from the config cache
if($instance.TypeName -eq $timerServiceInstanceName)
{
[string]$serverName = $server.Name

Write-Host -foregroundcolor DarkGray -NoNewline "Deleting xml files from config cache on server: "
Write-Host -foregroundcolor Gray $serverName

# Remove all xml files recursive on an UNC path
$path = "\\" + $serverName + "\c$\ProgramData\Microsoft\SharePoint\Config\*-*\*.xml"
Remove-Item -path $path -Force

break
}
}
}

Write-Host ""
}

#<summary>
# Clears the SharePoint cache on an UNC path
#</summary>
#<param name="$farm">The SharePoint farm object.</param>
function ClearTimerCache($farm)
{
Write-Host ""
Write-Host -foregroundcolor DarkGray "Clear the cache"

[string] $path = ""

# Iterate through each server in the farm, and each service in each server
foreach($server in $farm)
{
foreach($instance in $server.ServiceInstances)
{
# If the server has the timer service then force the cache settings to be refreshed
if($instance.TypeName -eq $timerServiceInstanceName)
{
[string]$serverName = $server.Name

Write-Host -foregroundcolor DarkGray -NoNewline "Clearing timer cache on server: "
Write-Host -foregroundcolor Gray $serverName

# Clear the cache on an UNC path
# 1 = refresh all cache settings
$path = "\\" + $serverName + "\c$\ProgramData\Microsoft\SharePoint\Config\*-*\cache.ini"
Set-Content -path $path -Value "1"

break
}
}
}

Write-Host ""
}

#**************************************************************************************
# Main script block
#**************************************************************************************

# Load SharePoint Powershell Snapin
Load-SharePoint-Powershell

# Get the local farm instance
$farm = Get-SPServer | where {$_.Role -match "Application"}

# Stop the SharePoint Timer Service on each server in the farm
StopSharePointTimerServicesInFarm $farm

# Delete all xml files from cache config folder on each server in the farm
DeleteXmlFilesFromConfigCache $farm

# Clear the timer cache on each server in the farm
ClearTimerCache $farm

# Start the SharePoint Timer Service on each server in the farm
StartSharePointTimerServicesInFarm $farm

Answer 2

I had this issue and it was driving me nuts, I made the same changes outlined in the question but it made no difference.

I did resolve the problem in the end and it appears that it was caused by Anti-Virus (System Center Endpoint Protection).

I had added the AV exclusions as advised in this article https://support.microsoft.com/en-us/help/952167/certain-folders-may-have-to-be-excluded-from-antivirus-scanning-when-y but it made no difference.

It turned out that SCEP was ignoring the exclusions for 'real-time protection'. Switching off real-time protection resolved the crawl issues for me.