Powershell Throttle Multi thread jobs via job completion

Question

All the tuts I have found use a pre defined sleep time to throttle jobs. I need the throttle to wait until a job is completed before starting a new one. Only 4 jobs can be running at one time.

So The script will run up 4 and currently pauses for 10 seconds then runs up the rest. What I want is for the script to only allow 4 jobs to be running at one time and as a job is completed a new one is kicked off.

Jobs are initialised via a list of servers names.

Is it possible to archive this?

$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"

$scriptBlock = { #DO STUFF }


$MaxThreads = 4

foreach($server in $servers) {
     Start-Job -ScriptBlock $scriptBlock -argumentlist  $server 
     While($(Get-Job -State 'Running').Count -ge $MaxThreads) {
          sleep 10 #Need this to wait until a job is complete and kick off a new one.
     }
}
Get-Job | Wait-Job | Receive-Job

Answer 1

You can test the following :

$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"
$scriptBlock = { #DO STUFF }
invoke-command -computerName $servers -scriptblock $scriptBlock -jobname 'YourJobSpecificName' -throttlelimit 4 -AsJob

This command uses the Invoke-Command cmdlet and its AsJob parameter to start a background job that runs a scriptblock on numerous computers. Because the command must not be run more than 4 times concurrently, the command uses the ThrottleLimit parameter of Invoke-Command to limit the number of concurrent commands to 4.

Be careful that the file contains the computer names in a domain.

Answer 2

In order to avoid inventing a wheel I would recommend to use one of the existing tools.

One of them is the script Invoke-Parallel.ps1. It is written in PowerShell, you can see how it is implemented directly. It is easy to get and it does not require any installation for using it.

Another one is the module SplitPipeline. It may work faster because it is written in C#. It also covers some more use cases, for example slow or infinite input, use of initialization and cleanup scripts.

In the latter case the code with 4 parallel pipelines will be

$servers | Split-Pipeline -Count 4 {process{ <# DO STUFF on $_ #> }}

Answer 3

I wrote a blog article which covers multithreading any given script via actual threads. You can find the full post here:

http://www.get-blog.com/?p=189

The basic setup is:

$ISS = [system.management.automation.runspaces.initialsessionstate]::CreateDefault()
$RunspacePool = [runspacefactory]::CreateRunspacePool(1, $MaxThreads, $ISS, $Host)
$RunspacePool.Open()

$Code = [ScriptBlock]::Create($(Get-Content $FileName))
$PowershellThread = [powershell]::Create().AddScript($Code)

$PowershellThread.RunspacePool = $RunspacePool
$Handle = $PowershellThread.BeginInvoke()
$Job = "" | Select-Object Handle, Thread, object
$Job.Handle = $Handle
$Job.Thread = $PowershellThread
$Job.Object = $Object.ToString()

$Job.Thread.EndInvoke($Job.Handle)
$Job.Thread.Dispose()

Answer 4

Instead of sleep 10 you could also just wait on a job (-any job):

Get-Job | Wait-Job -Any | Out-Null

When there are no more jobs to kick off, start printing the output. You can also do this within the loop immediately after the above command. The script will receive jobs as they finish instead of waiting until the end.

Get-Job -State Completed | % {
   Receive-Job $_ -AutoRemoveJob -Wait
}

So your script would look like this:

$servers = Get-Content "C:\temp\flashfilestore\serverlist.txt"

$scriptBlock = { #DO STUFF }

$MaxThreads = 4

foreach ($server in $servers) {
   Start-Job -ScriptBlock $scriptBlock -argumentlist $server 
   While($(Get-Job -State Running).Count -ge $MaxThreads) {
      Get-Job | Wait-Job -Any | Out-Null
   }
   Get-Job -State Completed | % {
      Receive-Job $_ -AutoRemoveJob -Wait
   }
}
While ($(Get-Job -State Running).Count -gt 0) {
   Get-Job | Wait-Job -Any | Out-Null
}
Get-Job -State Completed | % {
   Receive-Job $_ -AutoRemoveJob -Wait
}

---

Having said all that, I prefer runspaces (similar to Ryans post) or even workflows if you can use them. These are far less resource intensive than starting multiple powershell processes.

Answer 5

Your script looks good, try and add something like

Write-Host ("current count:" + ($(Get-Job -State 'Running').Count) + " on server:" + $server)

after your while loop to work out whether the job count is going down where you wouldn't expect it.

Answer 6

I noticed that every Start-Job command resulted in an additional conhost.exe process in the task manager. Knowing this, I was able to throttle using the following logic, where 5 is my desired number of concurrent threads (so I use 4 in my -gt statement since I am looking for a count greater than):

while((Get-Process conhost -ErrorAction SilentlyContinue).Count -gt 4){Start-Sleep -Seconds 1}