What are some suggested LogParser queries to run to detect sources of high network traffic?

Question

In looking at the network in/out metrics for our AWS/EC2 instance, I would like to find the sources of the high network traffic occurrences.

I have installed up Log Parser Studio and run a few queries - primarily looking for responses that took a while:

SELECT TOP 10000 * FROM '[LOGFILEPATH]' WHERE time-taken > 1000

I am also targeting time spans that cover when the network in/out spikes have occurred:

SELECT TOP 20000 * FROM '[LOGFILEPATH]' 
WHERE [date] BETWEEN TIMESTAMP('2013-10-20 02:44:00', 'yyyy-MM-dd hh:mm:ss') 
AND TIMESTAMP('2013-10-20 02:46:00', 'yyyy-MM-dd hh:mm:ss')

One issue is that the log files are 2-7 gigs (targeting single files per query). In trying Log Parser Lizard, it crashed with an out of memory exception on large files (boo).

What are some other queries, and methodologies I should follow to identify the source of the high network traffic, which would hopefully help me figure out how to plug the hole?

Thanks.

Answer 1

One function that may be of particular use to you is the QUANTIZE() function. This allows you to aggregate stats for a period of time thus allowing you to see spikes in a given time period. Here is one query I use that allows me to see when we get scanned:

SELECT QUANTIZE(TO_LOCALTIME(TO_TIMESTAMP(date, time)), 900) AS LocalTime,
    COUNT(*) AS Hits,
    SUM(sc-bytes) AS TotalBytesSent,
    DIV(MUL(1.0, SUM(time-taken)), Hits) AS LoadTime,
    SQRROOT(SUB(DIV(MUL(1.0, SUM(SQR(time-taken))), Hits), SQR(LoadTime))) AS StandardDeviation
INTO '[OUTFILEPATH]'
FROM '[LOGFILEPATH]'
WHERE '[WHERECLAUSE]'
GROUP BY LocalTime 
ORDER BY LocalTime

I usually output this to a .csv file and then chart in Excel to visually see where a period of time is out of normal range. This particular query breaks things down to 15 min segments based on the 900 passed to QUANTIZE. The TotalBytesSent, LoadTime and StandardDeviation allow me to see other aberrations in downloaded content or response times.

Another thing to look at is the number of requests a particular client has made to your site. The following query can help identify scanning or DoS activity coming in:

SELECT
    DISTINCT c-ip as ClientIP,
    COUNT(*) as Hits,
    PROPCOUNT(*) as Percentage
INTO '[OUTFILEPATH]'
FROM '[LOGFILEPATH]'
WHERE '[WHERECLAUSE]'
GROUP BY ClientIP
HAVING (Hits > 50)
ORDER BY Percentage DESC

Adjusting the HAVING clause will set the minimum number of requests an IP will need to make before it shows up. Based on the activity and the WHERE clause, 50 may be too low. The PROPCOUNT() function gives a percentage of the overall value of a particular field. In this case, it gives the what percent a particular IP of all the requests made to the site. Typically this will surface the IP addresses of search engines as well, but those are pretty easy to weed out.

I hope that gives you some ideas on what you can do.