graphite-api icon indicating copy to clipboard operation
graphite-api copied to clipboard

unorderable types: tuple() < NoneType() (500 INTERNAL SERVER ERROR)

Open bert2002 opened this issue 5 years ago • 0 comments

Hi, running into an weird problem. I am collecting data with collectd and push it to graphite-api which has graphite-carbon as backend. Works wonderful for all metrics, except for this one.

Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"reader": "whisper", "start": 1559806831, "end": 1559807131, "path": "/var/lib/graphite/whisper/collectd/test-node-server-1/processes/ps_state-running.wsp", "metric_path": "collectd.test-node-server-1.processes.ps_state-running", "event": "fetch"}
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"event": "Exception on /render [POST]", "exception": "Traceback (most recent call last):\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1982, in wsgi_app\n    response = self.full_dispatch_request()\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1614, in full_dispatch_request\n    rv = self.handle_user_exception(e)\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1517, in handle_user_exception\n    reraise(exc_type, exc_value, tb)\n  File \"/usr/lib/python3/dist-packages/flask/_compat.py\", line 33, in reraise\n    raise value\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1612, in full_dispatch_request\n    rv = self.dispatch_request()\n  File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1598, in dispatch_request\n    return self.view_functions[rule.endpoint](**req.view_args)\n  File \"/usr/lib/python3/dist-packages/graphite_api/app.py\", line 375, in render\n    data_store = fetchData(context, paths)\n  File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in fetchData\n    (node, node.fetch(startTime, endTime)) for node in single_nodes]\n  File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in <listcomp>\n    (node, node.fetch(startTime, endTime)) for node in single_nodes]\n  File \"/usr/lib/python3/dist-packages/graphite_api/node.py\", line 28, in fetch\n    return self.reader.fetch(startTime, endTime)\n  File \"/usr/lib/python3/dist-packages/graphite_api/finders/whisper.py\", line 121, in fetch\n    cached_datapoints = self.carbonlink.query(self.real_metric_path)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 199, in query\n    results = self.send_request(request)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 230, in send_request\n    host = self.select_host(metric)\n  File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 160, in select_host\n    for node in self.hash_ring.get_nodes(key):\n  File \"/usr/lib/
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: python3/dist-packages/graphite_api/carbonlink.py\", line 108, in get_nodes\n    index = bisect.bisect_left(self
[ps_state-running.dump.txt](https://github.com/brutasse/graphite-api/files/3260621/ps_state-running.dump.txt)
.ring, search_entry) % self.ring_len\nTypeError: unorderable types: tuple() < NoneType()"}

I already deleted all datasets (stopped all processes before), but the problem seems to be persistent. I attached a dump of the data (from whisper-fetch). I don't see anything wrong and wondering what is happening. because I have 10s of other server which the same configuration and dont run into this problem. Also all other metrics in processes are working as expected. Any idea?

OS: Debian 9.9
graphite-api: 1.1.3-2+deb9u1
graphite-carbon: 0.9.15-1
grafana: 6.2.2

Many thanks, bert

bert2002 avatar Jun 06 '19 08:06 bert2002