graphite-kenshin
graphite-kenshin copied to clipboard
douban
graphite API sending invalid JSON "NaN" causing grafana not showing anything.
graphite-kenshin sends single NaN values in some cases for some metrics.
Like this: [NaN, 1561983540],
This causes grafana failing to parse JSON and not showing anything in the graph.
@DEvil0000 You can use transformNull function to replace null values with the value specified by default.
Yes, I know that I can use transformNull but null is no issue. The issue is one of the values is NaN - literally. TransformNull is not affecting NaN which should not be there in any case.
I see. Are you able to reproduce this issue? If possible, could you share us the versions of related packages installed on your system and a sample data that can reproduce the problem?
My setup: collectd sending cpu, ram, interface usage and others to rurouni/kenshin. I am using only one instance of rurouni/kenshin.
I did the following which caused it and I was able to reproduce it this way: restart rurouni/kenshin and restart collectd service.
I guess you need to restart exactly when you receive or write metric data. So I restarted both about 10 times at random within short time. One or some metric series at random then get one of this broken entries (in kenshin as it looks) so the API will provide NaN for this value (reproducable every request then). So I guess the issue is more on kenshin side then here but you could at least mask it here by only sending valid JSON without NaN. Guess running this on a VM with slow CPU and maybe slow IO might helped to cause it if its a race condition.