Statsd & Graphite statistics per given item, not time

Question

It is easy to process any metrics with statsd and graphite, assuming they are measured per timespan. As an example, it is easy to track number of request per second.

On the other hand, sometimes might be useful to track a metric based on given "base item". For example, I process a data set and I want to track the percentage of invalid fields, number of actions necessary to process the data set etc. I can easily see a result as: "we had 10 invalid values in data set per second" and "we process 100 data fields in average, per second", but I would rather see something like "in 100 fields, there are 10 invalid values".

The results are similar when processing these fields takes similar amount of time. However, if it varies (and especially if the time differs according to the nature of data), time-based statistic is slightly confusing and does not reflect what I want to see.

Any solutions how to set up statsd / Graphite to solve the issue I have mentioned?

Answer 1

Creating a more meaningful relationship of time-series data at the boundary of StatsD/Graphite is quite difficult because, as you alluded to in the question, the data (used for deriving the percentage) is only related by time and key.

That said, for this type of data I've set up "percentage graphs" using asPercent(). Like this:

asPercent(stats_counts.myapp.messages.{ignored,dropped,recycled},
 stats_counts.myapp.messages.received)

You could also consider pushing this down into your application and performing the calculation where you can better relate the data and sending the data to Statsd as a gauge.