Powershell to parse filezilla log

Question

I need to pull the date, time and filename out of a filezilla log so I can insert them into a database.

This is what I have so far:

Select-String "C:\Scripts\testing\SSIS\testfiles\fzs-2014-04-15.log" -Pattern "STOR" -AllMatches
Foreach($line in $lines){
$line = $line -replace "C:\Path-to-logs\filezilla.log.*\) ",""
    $line = $line -replace "[AM|PM].* STOR ",""
    $line -split " "
}

I get the following results:

C:\path-to-logs\filezilla.log:114:(003173) 4/15/2014 3:04:20 AM - cwwsfge (192.168.250)> STOR NewFileupload.TXT
C:\path-to-logs\filezilla.log.log:210:(000182) 4/15/2014 6:21:21 AM - refect(192.168.250)> STOR Testfile_20140415
C:\path-to-logs\filezilla.log.log:662:(000179) 4/15/2014 6:27:13 AM - refect (192.168.2)> STOR FreeMoney.txt

So how do I get that info once I have it in the foreach???

Thanks!!

Answer 1

Editing to match your changes above. I think you can still get away with one regex -split and then picking off the elements that you need. Try this:

Foreach($line in $lines){
$parts = $line  -split '\) | - |\> STOR |:\('
    $logdate = [DateTime]::Parse($parts[2])
    $filename = $parts[4]

    write-host "Filename: $filename"
    write-host "Filedate: $($logdate.Date.ToShortDateString())"
    write-host "Filetime: $($logdate.TimeOfDay)`n"
}

Basically this is matching ") ", " - ", "> STOR " or ":(" and splitting the line across those. If the format is consistent, you should have 5 elements in each pass. I added some code to pick off the filename and parse the date and it yields this for the values in the example above:

Filename: NewFileupload.TXT
Filedate: 4/15/2014
Filetime: 03:04:20

Filename: Testfile_20140415
Filedate: 4/15/2014
Filetime: 06:21:21

Filename: FreeMoney.txt
Filedate: 4/15/2014
Filetime: 06:27:13

Answer 2

Using the previous answers I built a script to output in a tab separated format.

$logpath = "C:\Program Files (x86)\FileZilla Server\Logs"

# combine all the logs into one file
gci "$logpath\*.log" | sort LastWriteTime | % {$(Get-Content $_)} | Set-Content "$logpath\combo.txt"

# scan lines only containing login,file upload, and download
$lines = Select-String "$logpath\combo.txt" -Pattern "> RETR|> STOR|230 Logged on" -AllMatches

Foreach($line in $lines){
    $parts = $line  -split '\) | - |\> |:\('
    $logdate = [DateTime]::Parse($parts[2])
    $filename = $parts[4].Replace("230 Logged","Logged")
    $user = $parts[3]

    $user,$filename,$logdate -join "`t" | Out-File -FilePath "$logpath\output.txt" -Append -Width 500
}

Answer 3

I've created a script to parse logs into a form that's simpler to work with / sharing here for others with similar requirements to the OP.

For the specific question, here's how this code would be used:

[string[]]$logs = @('C:\Scripts\testing\SSIS\testfiles\fzs-2014-04-15.log') # you could add more / get these via Get-ChildItem / etc
[PSCustomObject[]]$data =  $logs | 
    Get-FzLogData | 
    ForEach-Object {
        if($_.Msg -match '\s*STOR\s(?<Filename>.+)') {
            [PSCustomObject]@{
                FileName = $Matches['FileName']
                DateTime = $_.DateTime
            }
        }
    }
$data # just shows the data for now...

Full Code:

class FzLogEntry {
    [long]$SessionId
    [DateTime]$DateTime
    [string]$User
    [version]$ClientIp # I've used version assuming it's always IPv4... if that assumption's wrong we may have to amend to string
    [string]$Msg
    FzLogEntry(){}
}

Function Get-FzLogData {
    [OutputType('FzLogEntry[]')]
    [CmdletBinding()]
    Param (
        [Parameter(Mandatory = $true, ValueFromPipeline = $true)]
        [string[]]$Path
    )
    Process {
        foreach ($p in $Path) {
            Get-Content -Path $p | ConvertTo-FzLogEntry
        }
    }
}
Function ConvertTo-FzLogEntry {
    [OutputType('FzLogEntry')]
    [CmdletBinding(DefaultParameterSetName = 'Default')]
    Param (
        [Parameter(ParameterSetName = 'Default', Mandatory = $true, ValueFromPipeline = $true)]
        [Parameter(ParameterSetName = 'Unparsable', Mandatory = $true, ValueFromPipeline = $true)]
        [string[]]$Line
        ,
        [Parameter(ParameterSetName = 'Unparsable', Mandatory = $true)]  # I don't really have a need for this; but it may be helpful if debugging to know when lines are not processed / have a gap in your results...
        [Switch]$IncludDefaultForUnparsable
        ,
        [Parameter(ParameterSetName = 'Unparsable')]
        [FzLogEntry]$DefaultValue = $null
    )
    Begin {
        [string]$regexPattern = @'
^\(
(?<SessionId>\d+)
\)\s
(?<DateTime>\S+\s\S+)
\s\-\s
(?<User>(?:\([^\)]+\))|(?:\S+))
\s\(
(?<ClientIp>[^>]+)
\)>\s*
(?<Msg>.*)
$
'@ -replace '[\r\n]+', ''    
    }
    Process {
        foreach ($l in $Line) {
            if ($l -match $regexPattern) {
                $fiddle = $Matches
                $fiddle.Remove(0)
                $fiddle.DateTime = $fiddle.DateTime -replace '^(\d+)\/(\d+)\/(\d+)', '$3-$2-$1' #correct date format to something that PS can auto convert to a DateTime.  Assumes dd/mm/yyyy format in the logs
                [FzLogEntry]$fiddle
            } else {
                if ($IncludDefaultForUnparsable.IsPresent) {
                    $DefaultValue
                }
            }
        }
    }
}

Related Gist will be kept up to date should I make any changes to this function.