Deleted/Empty Graphite Whisper Files Automatically Re-Generating

Question

I am trying to delete some old graphite test whisper metrics without any success. I can delete the metrics by removing the files. (See: How to cleanup the graphite whisper's data? ) But, within a few seconds of blowing away the files they regenerate (they are empty of metrics and stay that way since nothing is creating new metrics in those files). I've tried stopping carbon (carbon-cache.py stop) before deleting the files, but when I restart carbon (carbon-cache.py --debug start &) they just come back.

How do I permanently delete these files/metics so they never come back?

Answer 1

Are you running statsd or something similar?

I had the same issue and it was because statsd was flushing the counters it had in memory after I deleted the whisper files. I recycled statsd and the files stay deleted now.

Hope this helps

Answer 2

By default, Statsd will continue to send 0 for counters it hasn't received in the previous flush period. This causes carbon to recreate the file.

Lets say we want to delete a counter called 'bad_metrics.sent' from Statsd. You can use the Statsd admin interface running on port 8126 by default:

$ telnet <server-ip> 8126
Trying <server-ip>...
Connected to <server-name>.
Escape character is '^]'.

Use 'help' to get a list of commands:

help
Commands: stats, counters, timers, gauges, delcounters, deltimers, delgauges, quit

You can use 'counters' to see a list of all counters:

counters
{ 'statsd.bad_lines_seen': 0,
  'statsd.packets_received': 0,
  'bad_metrics.sent': 0 }
END

Its the 'delcounters', 'deltimers', and 'delgauges' commands that remove metrics from statsd:

delcounters bad_metrics.sent
deleted: bad_metrics.sent
END

After removing the metric from Statsd, you can remove the whisper file associated with it. In this example case, that would be:

/opt/graphite/storage/whisper/bad_metrics/sent.wsp

or (in Ubuntu):

/var/lib/graphite/whisper/bad_metrics/sent.wsp

Answer 3

The newest StatsD version has an option to not send zeroes after flush anymore, but only what is actually sent to it. If you turn that one the whisper files shouldn't get recreated: https://github.com/etsy/statsd/blob/master/exampleConfig.js#L39

Answer 4

We aren't running statsd, but we do run carbon-aggregator which serves a similar purpose. Restarting it solved a similar problem.