My method to send values of performance counters to Graphite is very slow. What is the bottleneck? And how to improve?

Question

Below I have some code to get the values of instances of performance counters (which are instantiated once a page is visited) and send them to Graphite to display graphs in the following format:

[Path in Graphite (e.g., metric.pages.Counter1)] [value of counter] [epoch time]

To do this I made the following code where the writer is configured correctly to work:

# Get all paths to MultipleInstance counters and averages that start with "BLABLA" and 
# put them into an array and get the epoch time 
$pathsWithInstances = (get-counter -ListSet BLABLA*) | select -ExpandProperty PathsWithInstances
$epochtime = [int][double]::Parse((Get-Date -UFormat %s))

# This functions splits the path (e.g., \BLABLA Web(welcome)\Page Requests) into three 
# parts: the part before the 
# opening brace (the CounterCategory, e.g., "\BLABLA Web"), the part in between the braces 
# (the page or 
# service, e.g., "welcome"), and the part after the closing brace (the name of the test, 
# e.g., 
# "\Page Requests"). We obtain the metric out of this information and send it to 
# Graphite.
enter code here
foreach ($pathWithInstance in $pathsWithInstances)
{   
    $instanceProperties = $pathWithInstance.Split('()')
    $counterCategory = $instanceProperties[0]

    if ($counterCategory -eq ("\BLABLA Web") )
    {
        # Replace the * with nothing so that counters that are used to display the 
        # average (e.g., \BLABLAWeb(*)\Page Requests) are displayed on top in the  
        # Graphite directory.

        $pagePath = $instanceProperties[1].Replace('*','')
        $nameOfTheTest = $instanceProperties[2]

        # Countername which is used in Graphite path gets whitespace and backslash 
        # removed in the name used for the path in Graphite (naming conventions)
        $counterName = $nameOfTheTest.Replace(' ','').Replace('\','')
        $pathToPerfCounter = $pathWithInstance
        $pathInGraphite = "metrics.Pages." + $pagePath + $counterName

        #Invoked like this since otherwise the get-counter [path] does not seem to work
       $metricValue = [int] ((Get-Counter "$pathToPerfCounter").countersamples | select -
             property cookedvalue).cookedvalue           
        $metric = ($pathInGraphite + " " + $metricValue +  " " + $epochTime)

        $writer.WriteLine($metric) 
        $writer.Flush() 

    }

}

Unfortunately this code is very slow. It takes about one second for every counter to send a value. Does someone see why it is so slow and how it can be improved?

Answer 1

You're getting one counter at a time, and it takes a second for Get-Counter to get and "Cook" the values. Get-Counter will accept an array of counters, and will sample, "cook" and return them all in that same second. You can speed it up by sampling them all at once, and then parsing the values from the array of results:

$CounterPaths = (
 '\\Server1\Memory\Page Faults/sec',
 '\\Server1\Memory\Available Bytes'
 )


(Measure-Command {
foreach ($CounterPath in $CounterPaths)
{Get-Counter -counter $counterpath}
}).TotalMilliseconds

(Measure-Command {
 Get-Counter $CounterPaths
 }).TotalMilliseconds


2017.4693
1012.3012

Example:

foreach ($CounterSample in (Get-Counter $CounterPaths).Countersamples)
{
  "Path = $($CounterSample.path)"
  "Metric = $([int]$CounterSample.CookedValue)"
}

Path = \\Server1\memory\page faults/sec
Metric = 193
Path = \\Server1\memory\available bytes
Metric = 1603678208

Answer 2

Use the Start-Job cmdlet, to create separate threads for each counter.

Here is a simple example of how to take the Counter Paths and pass them into an asynchronous ScriptBlock:

$CounterPathList = (Get-Counter -ListSet Processor).PathsWithInstances.Where({ $PSItem -like '*% Processor Time' });

foreach ($CounterPath in $CounterPathList) {
    Start-Job -ScriptBlock { (Get-Counter -Counter $args[0]).CounterSamples.CookedValue; } -ArgumentList $CounterPath;
}

# Call Receive-Job down here, once all jobs are finished

IMPORTANT: The above example uses PowerShell version 4.0's "method syntax" for filtering objects. Please make sure you're running PowerShell version 4.0, or change the Where method to use the traditional Where-Object instead.