Splunk - chart 2 time periods on 1 report

Question

I'm trying to chart a comparison between 1 hour of tomcat logs over 2 weeks (same hour, for instance 2-3pm on Tuesday this week against 2-3pm pm Tuesday last week).

I have seen 2 solutions described.

• Described here - http://blogs.splunk.com/2013/01/17/two-time-series-one-chart-part-two/

This describes performing a single report across the entire time period and (effectively) discarding results falling outside the periods I'm interested in. (OK, it more describes comparing to contiguous time periods, but you can work out how to discard the intervening ignored results).

• (Can't remember a link to this, but effectively appending searches across the time periods together and manipulating the _time for the one period:

sourcetype="ws-logs" source="/var/local/catalina/logs/localhost_access_log.*" 
    "/importantCall" AND httpStatusCode>=200 AND httpStatusCode<300 
    earliest=-60m@h latest=-0m@m 
| eval marker="today" 
| append [search 
    sourcetype="ws-logs" source="/var/local/catalina/logs/localhost_access_log.*" 
        "/track/sale" AND httpStatusCode>=200 AND httpStatusCode<300 
        earliest=-10140m@h latest=-10080m@m 
    | eval marker="weekAgo" 
    | eval w1_time=_time+(7*24*60*60)] 
| eval _time=if(isnotnull(w1_time), w1_time, _time) 
| chart 
       count(eval(marker=="today")) as lastHour
     , count(eval(marker=="weekAgo")) as sameTimeLastWeek
    by _time span=10m 
| rename _time AS Time 
| eval Time=strftime(Time, "%H:%M") 

Just to explain here, I snap-to the start of the previous hour for the start of time period, so you may see more than 1 hour's data. And I format the time axis of the chart to only show the hour/minute as showing the date is wrong.

OK, now due to the volume of data, the report on the entire time period is a non-runner, too much data (especially since I'm discarding 98.8% of it [keeping 120 minutes out of a total of 10200 minutes of data]).

The second search works correctly and charts the data nicely.

However, if I schedule this search, only the initial search gets run, NOT the appended one. Does anyone know of a better solution or what I need to do to get the scheduled searches appending correctly?

Thanks

Results if I run the report from the 'Search' app:

   Time   lastHour  sameTimeLastWeek
1  13:00  35        43
2  13:10  50        47
3  13:20  72        50
4  13:30  75        38
5  13:40  108       51
6  13:50  100       32
7  14:00  24        11
8  14:10  47        32
9  14:20  38        56

Results of a scheduled search over the same period:

   Time   lastHour  sameTimeLastWeek
1  13:00  35        0
2  13:10  50        0
3  13:20  72        0
4  13:30  75        0
5  13:40  108       0
6  13:50  100       0
7  14:00  24        0
8  14:10  47        0
9  14:20  38        0

Answer 1

First, try not to use "append" when you're getting a lot of data. It's results are truncated at some point, I believe the default is 50,000 events. Check the docs for limits.conf.

Here's the other blog I think you meant to mention. It was recently updated to mention a better solution in the Exploring Splunk book, which is full text available online and is mentioned at the end of the post. This blog posting does an excellent job of explaining the concepts, with the exception of the "append" limitation above:

http://blogs.splunk.com/2012/02/19/compare-two-time-ranges-in-one-report/

Or go right to the book, page 85, titled: Charting Week Over Week Results:

http://www.splunk.com/goto/book