Storing ever-increasing value in Graphite

Question

I am monitoring a queue to track the number of error messages over time in Graphite. The errored messages sit in an end state status of 'error' and thus the simple sending of the count to a gauge or counter does not work as expected.

To illustrate:

minute    path                result
---       ----                ------
1         queue.errors 3000   3000 errors (the baseline)
2         queue.errors 3002   2 errors 
3         queue.errors 3005   3 errors
4         queue.errors 3010   5 errors

Is there a way via statsd or graphite to handle this sort of metric? The queue count of errors will increase forever but the meaningful value is really only the change since the last value.

I've read about statsd's Gauge Delta which looks to support the opposite of what I need:

statsd.gauge('foo', 70)  # Set the 'foo' gauge to 70.
statsd.gauge('foo', 1, delta=True)  # Set 'foo' to 71.

But, what I really need is this:

statsd.gauge('foo', 70)  # Set the 'foo' gauge to 70.
statsd.gauge('foo', 71, ?????=True)  # Set 'foo' to 1.

Answer 1

Funny that you should ask, there is an ongoing thread about the ability to track absolute values in statsd and get the derivative. See https://github.com/etsy/statsd/issues/324

The gist of it is that statsd does not support counters as defined as an ever-increasing natural integer. Older tools, like rrdtool, do but they don't offer nearly the same flexibility as statsd.

Answer 2

If you want to get the rate of change from an gauge or counter value, just enclose it in either the derivative() or nonNegativeDerivate() functions.