Managing the running time of background jobs. Timing out if not completed after x seconds,

Question

I would like to time my background jobs (started with start-job) and time them out after x seconds. I find it hard however to keep track of the running time on each separate job (I am running aprox 400 jobs).

I wish there was a way to time out the job and set it to failed if not completed in X seconds, but I find no timeout-parameter.

What would be a good way to track the individual run-time of the jobs?

I guess I could create a hashtable with start-time of each job and the job-id and check against the running state and do a manual timeout, but that sounds kinda "inventing the wheel". Any ideas?

Edit Thank you everyone for a fruitful discussion and great inspiration on this topic!

Answer 1

You can use a hash table of timers:

 $jobtimer = @{}

 foreach ($job in $jobs){
   start-job -name $job -ScriptBlock {scriptblock commands}
   $jobtimer[$job] = [System.Diagnostics.Stopwatch]::startnew()
   }

The running time of each job will be in $jobtimer[$job].elapsed

Answer 2

Just walk through the list of running jobs and stop any that have run past your timeout spec e.g.:

$timeout = [timespan]::FromMinutes(1)
$now = Get-Date
Get-Job | Where {$_.State -eq 'Running' -and 
                 (($now - $_.PSBeginTime) -gt $timeout)} | Stop-Job

BTW there are more properties to a job object than the default formatting shows e.g.:

3 >  $job | fl *


State         : Running
HasMoreData   : True
StatusMessage :
Location      : localhost
Command       :  Start-sleep -sec 30
JobStateInfo  : Running
Finished      : System.Threading.ManualResetEvent
InstanceId    : de370ea8-763b-4f3b-ba0e-d45f402c8bc4
Id            : 3
Name          : Job3
ChildJobs     : {Job4}
PSBeginTime   : 3/18/2012 11:07:20 AM
PSEndTime     :
PSJobType     : BackgroundJob
Output        : {}
Error         : {}
Progress      : {}
Verbose       : {}
Debug         : {}
Warning       : {}

Answer 3

You can specify the timeout option of Wait-Job:

-Timeout

Determines the maximum wait time for each background job, in seconds. The default, -1, waits until the job completes, no matter how long it runs. The timing starts when you submit the Wait-Job command, not the Start-Job command.

If this time is exceeded, the wait ends and the command prompt returns, even if the job is still running. No error message is displayed.

Here's some example code:

This part just makes some test jobs:

Remove-Job -Name *
$jobs = @()
1..10 | % {
    $jobs += Start-Job -ScriptBlock {
        Start-Sleep -Seconds (Get-Random -Minimum 5 -Maximum 20)
    }
}

The variable $timedOutJobs contains jobs that timed out. You can then restart them or what have you.

$jobs | Wait-Job -Timeout 10 
$timedOutJobs = Get-Job | ? {$_.State -eq 'Running'} | Stop-Job -PassThru

Answer 4

For completeness this answer combines the maximum seconds per job and the maximum concurrent jobs running. As this is what most people are after.

The example below retrieves the printer configuration for each print server. There can be over 3000 printers, so we added throttling.

$i = 0
$maxConcurrentJobs = 40
$maxSecondsPerJob = 60
$jobTimer = @{ }

$StopLongRunningJobs = {
    $jobTimer.GetEnumerator().where( {
            ($_.Value.IsRunning) -and
            ($_.Value.Elapsed.TotalSeconds -ge $maxSecondsPerJob)
        }).Foreach( {
            $_.Value.Stop()
            Write-Verbose "Stop job '$($_.Name.Name)' that ran for '$($_.Value.Elapsed.TotalSeconds)' seconds"
            Stop-Job $_.Name
        })
}

Foreach ($Computer in @($GetPrinterJobResults.Where( { $_.Data }) )) {
    foreach ($Printer in $Computer.Data) {
        do {
            & $StopLongRunningJobs
            $running = @(Get-Job -State Running)
            $Wait = $running.Count -ge $maxConcurrentJobs

            if ($Wait) {
                Write-Verbose 'Waiting for jobs to fininsh'
                $null = $running | Wait-Job -Any -Timeout 5
            }
        } while ($Wait)

        $i++
        Write-Verbose "$I $($Computer.ComputerName) Get print config '$($Printer.Name)'"
        $Job = $Printer | Get-PrintConfiguration -AsJob -EA Ignore
        $jobtimer[$Job] = [System.Diagnostics.Stopwatch]::StartNew()
    }
}

$JobResult = Get-Job | Wait-Job -Timeout $maxSecondsPerJob -EA Ignore
$JobResult = Get-Job | Receive-Job -EA Ignore
$JobResult.count