Backfill/Rebalance process creates spikes on counters

[rsommer]

We recently did a cluster migration within our graphite setup and used bucky rebalance to adjust data distribution afterwards. After the migration was done, we discoverd spikes in our counter-based graphs, all appearing at the hour-boundary. These spikes appeared only after the migration and only for backfilled counter data - all newly written datapoints are spike-less as they have been before. Example graph from our dashboards:

All the spikes have not been there before rebalancing took place and are gone for all datapoints that have been written since.

This was done using the last github release 0.3.2.

[deniszh]

Hi @rsommer

Do you using lock in your carbon / go-carbon? You can also try our fork of buckytools, we're using it in prod w/o any issues. But please use master version, we're not doing releases often.

[rsommer]

Hi @deniszh ,

indeed, flock was at its default value (false) - under normal conditions there is only go-carbon writing to the whisper files. Strange behaviour, though - because only Counts seem to be affected. I'll retry some cluster layout changes using your fork and flock set to true if I find some time.