graphite-api
graphite-api copied to clipboard
unorderable types: tuple() < NoneType() (500 INTERNAL SERVER ERROR)
Hi, running into an weird problem. I am collecting data with collectd and push it to graphite-api which has graphite-carbon as backend. Works wonderful for all metrics, except for this one.
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"reader": "whisper", "start": 1559806831, "end": 1559807131, "path": "/var/lib/graphite/whisper/collectd/test-node-server-1/processes/ps_state-running.wsp", "metric_path": "collectd.test-node-server-1.processes.ps_state-running", "event": "fetch"}
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: {"event": "Exception on /render [POST]", "exception": "Traceback (most recent call last):\n File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1982, in wsgi_app\n response = self.full_dispatch_request()\n File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1614, in full_dispatch_request\n rv = self.handle_user_exception(e)\n File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1517, in handle_user_exception\n reraise(exc_type, exc_value, tb)\n File \"/usr/lib/python3/dist-packages/flask/_compat.py\", line 33, in reraise\n raise value\n File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1612, in full_dispatch_request\n rv = self.dispatch_request()\n File \"/usr/lib/python3/dist-packages/flask/app.py\", line 1598, in dispatch_request\n return self.view_functions[rule.endpoint](**req.view_args)\n File \"/usr/lib/python3/dist-packages/graphite_api/app.py\", line 375, in render\n data_store = fetchData(context, paths)\n File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in fetchData\n (node, node.fetch(startTime, endTime)) for node in single_nodes]\n File \"/usr/lib/python3/dist-packages/graphite_api/render/datalib.py\", line 167, in <listcomp>\n (node, node.fetch(startTime, endTime)) for node in single_nodes]\n File \"/usr/lib/python3/dist-packages/graphite_api/node.py\", line 28, in fetch\n return self.reader.fetch(startTime, endTime)\n File \"/usr/lib/python3/dist-packages/graphite_api/finders/whisper.py\", line 121, in fetch\n cached_datapoints = self.carbonlink.query(self.real_metric_path)\n File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 199, in query\n results = self.send_request(request)\n File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 230, in send_request\n host = self.select_host(metric)\n File \"/usr/lib/python3/dist-packages/graphite_api/carbonlink.py\", line 160, in select_host\n for node in self.hash_ring.get_nodes(key):\n File \"/usr/lib/
Jun 06 15:45:31 monitoring-1 gunicorn3[725]: python3/dist-packages/graphite_api/carbonlink.py\", line 108, in get_nodes\n index = bisect.bisect_left(self
[ps_state-running.dump.txt](https://github.com/brutasse/graphite-api/files/3260621/ps_state-running.dump.txt)
.ring, search_entry) % self.ring_len\nTypeError: unorderable types: tuple() < NoneType()"}
I already deleted all datasets (stopped all processes before), but the problem seems to be persistent. I attached a dump of the data (from whisper-fetch). I don't see anything wrong and wondering what is happening. because I have 10s of other server which the same configuration and dont run into this problem. Also all other metrics in processes
are working as expected. Any idea?
OS: Debian 9.9
graphite-api: 1.1.3-2+deb9u1
graphite-carbon: 0.9.15-1
grafana: 6.2.2
Many thanks, bert