Export-CSV cmdlet rewriting entire CSV during each iteration of a FOREACH statement

Question

I'm working with some code that is going to take a series of performance counters, and then put the counters in a .csv file that rolls over every time it hits 1MB.

$Folder="C:\Perflogs\BBCRMLogs" # Change the bit in the quotation marks to whatever directory you want the log file stored in

$Computer = $env:COMPUTERNAME
$1GBInBytes = 1GB
$p = LOTS OF COUNTERS;

# If you want to change the performance counters, change the above list. However, these are the recommended counters for a client machine. 

$num  = 0
$file = "$Folder\SQL_log_${num}.csv"

if( !(test-path $folder)) {New-Item $Folder -type directory}

Get-Counter -counter $p -SampleInterval 2 -Continuous | Foreach {

    if ((Get-Item $file -ErrorAction SilentlyContinue ).Length -gt 1mb) 
    {
        $num +=1
        $file = "$Folder\SQL_log_${num}.csv"
    }

    $_

} | Foreach-Object { $_ | Export-Csv $file -Force -Append}

Right now, it's working quite well. The iteration works fine, and it does create a new file each time the .csv reaches 1MB. However, each .CSV after the first is being created after 2 minutes already at 1MB, causing a new file to be created. I'm not quite sure why this is occurring, although I believe it's because Powershell is just rewriting the entirety of the .csv each time it creates it.

Answer 1

[I'm posting this as a new answer rather than editing the original because it's completely different. Replacing or appending to the original answer would make the ensuing discussion confusing.]

What you need to do is use a regex to extract the values from the Readings property of the output of Get-Counter, and manually construct CSV output from the timestamp and those values. Change the last line to this (format according to your preferred style):

| %{'"' + (Get-Date $_.Timestamp -f 's') + '","' + (([regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') | select -ExpandProperty Value) -join '","') + '"'} | Out-File $file -Append -Encoding ASCII

To break that down:

• (Get-Date $_.Timestamp -f 's') This part is not strictly necessary, though I think it will make your results easier to follow. The 's' format puts the date in an ISO 8601 sortable pattern. You could substitute 'u' for another sortable format, or use your favorite custom format string. Or just replace it with $_.Timestamp to retain the original format.
• [regex]::matches($_.Readings, '(?<=\\\\.+?:\n)(.+?)(?=\n)') The regex matches the contents of any line that is preceded by a line that begins with \\ and ends with : (those pesky counter names you wanted to get rid of). Note that I'm using [regex]::matches, which performs a global match, as opposed to [regex]::match or -match, which will just give you the first match for each string (the Readings property is a single string, so only the first counter reading would be returned).
• | select -ExpandProperty Value Produces an array of all the matches, which you can then join with "," and surround with "'s to produce CSV output.

Since you're not using a conversion function, you also need to construct a header row. Add this line right above the pipeline:

`'"Timestamp","' + ($p -join '","') + '"' | Out-File $file -Append -Encoding ASCII`

That's assuming that $p is an array (which it should be). If it's a string, then depending on the format you can either use it as-is, or -split it and rejoin it in CSV format.

Answer 2

Change the last line to this, to convert each line to CSV format and then append it to the output file:

} | Foreach-Object {($_ | ConvertTo-Csv -NoTypeInformation)[1] | Out-File $file -Append -Encoding ASCII}.

A few notes:

• The -Encoding ASCII is not strictly necessary, but you might have trouble with a Unicode CSV file in some applications (Excel, for example, won't open it as a CSV file by default, and everything will be in Column A)
• The reason for the index in ($_ | ConvertTo-Csv -NoTypeInformation)[1] is that ConvertTo-Csv -NoTypeInformation still outputs the header row each time, so you want to grab the second line of the two-line output (($_ | ConvertTo-Csv -NoTypeInformation)[0] is the header row)
• Since you're not outputting a header row, you'll need to output one to $file before the loop