highestAverage fail with Internal Server Error when some matching whisper databases contains "None" values

[bplessis]

Hi,

I'm not sure if everything should be fixed on the "graphite-api" side and will try to also fix the data-collector (netapp-harvest) but i'm not sure sending back an exception trace is really what should be done.

Here is the problem as i see it, the data collection sometime doesn't record values, probably because lack of change on the specific part of the monitored system, so a "None" is recorded in the whisper database.

When requesting/graphing (with grafana) the value using aliasByNode(netapp.perf.$Group.$Cluster.svm.$SVM.vol.*.read_data, 7) no problem occur but the default graphs provided with the tool have a 'highestAverage' filter on to limit the subset to the "biggest" consumers:

aliasByNode(highestAverage(netapp.perf.$Group.$Cluster.svm.$SVM.vol.*.read_data, $TopResources), 7)

When adding highestAverage the query give back something like that:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/flask/app.py", line 1982, in wsgi_app response = self.full_dispatch_request() File "/usr/lib/python3/dist-packages/flask/app.py", line 1614, in full_dispatch_request rv = self.handle_user_exception(e) File "/usr/lib/python3/dist-packages/flask/app.py", line 1517, in handle_user_exception reraise(exc_type, exc_value, tb) File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise raise value File "/usr/lib/python3/dist-packages/flask/app.py", line 1612, in full_dispatch_request rv = self.dispatch_request() File "/usr/lib/python3/dist-packages/flask/app.py", line 1598, in dispatch_request return self.view_functionsrule.endpoint File "/usr/lib/python3/dist-packages/graphite_api/app.py", line 415, in render series_list = evaluateTarget(context, target, data_store) File "/usr/lib/python3/dist-packages/graphite_api/evaluator.py", line 71, in evaluateTarget result = evaluateTokens(requestContext, tokens, data_store) File "/usr/lib/python3/dist-packages/graphite_api/evaluator.py", line 103, in evaluateTokens replacements) File "/usr/lib/python3/dist-packages/graphite_api/evaluator.py", line 131, in evaluateTokens for arg in tokens.call.args] File "/usr/lib/python3/dist-packages/graphite_api/evaluator.py", line 131, in for arg in tokens.call.args] File "/usr/lib/python3/dist-packages/graphite_api/evaluator.py", line 103, in evaluateTokens replacements) File "/usr/lib/python3/dist-packages/graphite_api/evaluator.py", line 137, in evaluateTokens ret = func(requestContext, *args, **kwargs) File "/usr/lib/python3/dist-packages/graphite_api/functions.py", line 2279, in highestAverage return sorted(seriesList, key=safeAvg)[-n:] TypeError: unorderable types: float() < NoneType()

I was using debian-strech packaged 1.1.3, i also tried the latest master branch this didn't fix the issue.

[bplessis]

look like i can hack around most of the time by inserting maximumAbove() function in the path, quitte cumbersome however :/

[cflinspach]

Have you tried keepLastValue?

[pkittenis]

nonullpoints can also be used to filter out null datapoints.

[bplessis]

Not tested any of thoses, however by definition keepLastValue would change the meaning of the data.

As for noNullPoints it look like a filter on the data output'd by the api, while the exception occur when sorting data inside the function (also don't really know how to insert this in grafana).

A few intern functions seems protected from null value (including maximumAbove(), why not highestAverage() ?

[pkittenis]

It's a bug in the code, yes. There are a couple functions that could help to work around.

A PR would not be too difficult to make, however, there are lots of ready PRs that have yet to be reviewed so a work around or self-modification are the best options so far (not speaking as member of the project).