librenms
librenms copied to clipboard
librenms
Poller stops working if redis reboots for any reason
The problem
Hello,
While making some resilience tests, we discovered that if for any reason the redis server was rebooted, LibreNMS pollers just stop working. And do not recover without a reboot.
As soon as the redis reboot occurs, the pollers try to reconnect during half a second then definitely stop.
The redis version used for the tests was 5.0.7. Only 10% of the logs are provided, the github form does not allow more.
Output of ./validate.php
====================================
Component | Version
--------- | -------
LibreNMS | 22.7.0
DB Schema | 2022_07_19_081224_plugins_unique_index (243)
PHP | 8.0.22
Python | 3.10.5
MySQL | 8.0.30-0ubuntu0.20.04.2
RRDTool | 1.7.2
SNMP | 5.9.3
====================================
[OK] Installed from the official Docker image; no Composer required
[OK] Database connection successful
[OK] Database Schema is current
[OK] SQL Server meets minimum requirements
[OK] lower_case_table_names is enabled
[OK] MySQL engine is optimal
[OK]
[OK] Database schema correct
[OK] MySQl and PHP time match
[OK] Connected to rrdcached
[INFO] Detected Dispatcher Service
[WARN] IPv6 is disabled on your server, you will not be able to add IPv6 devices.
[OK] rrdtool version ok
[OK] Connected to rrdcached
[WARN] Updates are managed through the official Docker image
What was the last working version of LibreNMS?
No response
Anything in the logs that might be useful for us?
2022-08-23T13:34:55.623503001Z Services_0-6(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.623572363Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.623593366Z Services_0-6(ERROR):Services poller exception! Connection closed by server.
2022-08-23T13:34:55.625614276Z Poller_0-15(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.625703558Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.625769704Z Poller_0-15(ERROR):Poller poller exception! Connection closed by server.
2022-08-23T13:34:55.626633427Z Poller_0-14(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.626695471Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.626821013Z Poller_0-14(ERROR):Poller poller exception! Connection closed by server.
2022-08-23T13:34:55.627817159Z Alerting(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.627848582Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.627859617Z Alerting(ERROR):Alerting poller exception! Connection closed by server.
2022-08-23T13:34:55.628546723Z Services_0-4(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.628582340Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.628659685Z Services_0-4(ERROR):Services poller exception! Connection closed by server.
2022-08-23T13:34:55.629649320Z Poller_0-12(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.629709828Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.629719731Z Poller_0-12(ERROR):Poller poller exception! Connection closed by server.
2022-08-23T13:34:55.630502885Z Discovery_0-5(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.630532795Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.630540582Z Discovery_0-5(ERROR):Discovery poller exception! Connection closed by server.
2022-08-23T13:34:55.631028885Z Poller_0-11(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.631046102Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.631092444Z Poller_0-11(ERROR):Poller poller exception! Connection closed by server.
2022-08-23T13:34:55.631789787Z Discovery_0-10(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.631815917Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.631870522Z Discovery_0-10(ERROR):Discovery poller exception! Connection closed by server.
2022-08-23T13:34:55.633122723Z Discovery_0-13(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.633142603Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.633199763Z Discovery_0-13(ERROR):Discovery poller exception! Connection closed by server.
2022-08-23T13:34:55.634228653Z Poller_0-9(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.634258260Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.644599848Z Discovery_0-1(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.644644085Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.670283730Z Poller_0-23(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.670339987Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.682387074Z Discovery_0-11(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.682435635Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.684939503Z Poller_0-19(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.684992542Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.685033245Z Poller_0-19(ERROR):Poller poller exception! Connection closed by server.
2022-08-23T13:34:55.687810644Z Discovery_0-7(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.687864730Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.697599709Z Poller_0-18(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.698003143Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.699376842Z Traceback (most recent call last):
2022-08-23T13:34:55.699569542Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.699707484Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.699799047Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.699823341Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.699832811Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.699841384Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.699849691Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.699856694Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.699861620Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.699940077Z fail(error)
2022-08-23T13:34:55.702988944Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.703025722Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.705715234Z Traceback (most recent call last):
2022-08-23T13:34:55.710244657Z Traceback (most recent call last):
2022-08-23T13:34:55.710590916Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.710623298Z raise error
2022-08-23T13:34:55.710652280Z Discovery_0-2(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.710663403Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.710669520Z Traceback (most recent call last):
2022-08-23T13:34:55.710674434Z Traceback (most recent call last):
2022-08-23T13:34:55.710767984Z Traceback (most recent call last):
2022-08-23T13:34:55.710922862Z Traceback (most recent call last):
2022-08-23T13:34:55.711285198Z Traceback (most recent call last):
2022-08-23T13:34:55.711325239Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.711352307Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.711358786Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.711364564Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.711689143Z Traceback (most recent call last):
2022-08-23T13:34:55.711706381Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.711712047Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.711716885Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.711723090Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.711728202Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.711770874Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.711799224Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.711824732Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.711831644Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.711837726Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.711843515Z Traceback (most recent call last):
2022-08-23T13:34:55.711998607Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.712017827Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.712024213Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.712030557Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.712036370Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.712042566Z return do()
2022-08-23T13:34:55.712063359Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.712074797Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.712107584Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.712120294Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.712143793Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.712154026Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.712159845Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.712165137Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.712181642Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.712192717Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.712248315Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.712267069Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.712274007Z Traceback (most recent call last):
2022-08-23T13:34:55.712279777Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.712285727Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.712681209Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.712702703Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.712711360Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.712733077Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.712739877Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.712747706Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.712995455Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.713019127Z fail(error)
2022-08-23T13:34:55.713025950Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.713032756Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.713038758Z Discovery_0-16(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.713044870Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.713050573Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.713056581Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.713066757Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.713072687Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.713078619Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.713084425Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.713109447Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.713120882Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.713126705Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.713131940Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.713352840Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.713372295Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.713378987Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.713385034Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.713390638Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.713396833Z fail(error)
2022-08-23T13:34:55.713403873Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.713409610Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.713416957Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.713422919Z raise error
2022-08-23T13:34:55.713482695Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.713494258Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.713530725Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.713541111Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.713547512Z Discovery_0-16(ERROR):Discovery poller exception! Connection closed by server.
2022-08-23T13:34:55.713653529Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.713672959Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.713678971Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.713684925Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.713689964Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.713695389Z fail(error)
2022-08-23T13:34:55.713840733Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.713859247Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.713865477Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.713871703Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.713877106Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.713883554Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.713889316Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.713895146Z fail(error)
2022-08-23T13:34:55.713929256Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.713950965Z fail(error)
2022-08-23T13:34:55.713957253Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.713962975Z raise error
2022-08-23T13:34:55.713968061Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.713974138Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.714329832Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.714353644Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.714360540Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.714366937Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.714479929Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.714499320Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.714519871Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.714570576Z return do()
2022-08-23T13:34:55.714604324Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.714628093Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.714795338Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.714813020Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.714819397Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.714825323Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.714870255Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.714881958Z raise error
2022-08-23T13:34:55.714888106Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.714893576Z return do()
2022-08-23T13:34:55.714899338Z Poller_0-7(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.714904751Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.714910270Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.714915624Z raise error
2022-08-23T13:34:55.714920674Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.714925626Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.714930463Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.714935452Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.715010801Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.715033477Z fail(error)
2022-08-23T13:34:55.715070317Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.715077050Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.715083413Z Traceback (most recent call last):
2022-08-23T13:34:55.715269749Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.715287369Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.715295266Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.715301266Z fail(error)
2022-08-23T13:34:55.715520794Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.715547706Z raise error
2022-08-23T13:34:55.715554407Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.715574008Z return do()
2022-08-23T13:34:55.715625046Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.715638820Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.715644586Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.715650349Z return do()
2022-08-23T13:34:55.715655153Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.715660149Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.715665588Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.715671639Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.715691273Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.715696879Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.715740842Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.715761426Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.715840770Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.715852554Z return do()
2022-08-23T13:34:55.715857836Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.715863299Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.715868562Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.715873438Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.715879649Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.715884796Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.715889726Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.715894732Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.715945642Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.715958646Z response = connection.read_response()
2022-08-23T13:34:55.715973536Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.715981621Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.716076222Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.716095293Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.716100937Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.716131482Z response = connection.read_response()
2022-08-23T13:34:55.716180360Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.716194834Z response = connection.read_response()
2022-08-23T13:34:55.716202436Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.716209831Z fail(error)
2022-08-23T13:34:55.716219758Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.716228052Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.716235056Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.716272825Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.716280371Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.716285489Z raw = self._buffer.readline()
2022-08-23T13:34:55.716290133Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.716295073Z raise error
2022-08-23T13:34:55.716308959Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.716324298Z fail(error)
2022-08-23T13:34:55.716356211Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.716369114Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.716488623Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.716501864Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.716510285Z Poller_0-9(ERROR):Poller poller exception! Connection closed by server.
2022-08-23T13:34:55.716518084Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.716526393Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.716579278Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.716591790Z raise error
2022-08-23T13:34:55.716598442Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.716604911Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.716665384Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.716674115Z raise error
2022-08-23T13:34:55.716678906Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.716684036Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.716702392Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.716707994Z fail(error)
2022-08-23T13:34:55.716730857Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.716745207Z raw = self._buffer.readline()
2022-08-23T13:34:55.716783336Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.716794706Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.716802696Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.716810432Z response = connection.read_response()
2022-08-23T13:34:55.716881793Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.716891413Z return do()
2022-08-23T13:34:55.717178434Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.717208643Z response = connection.read_response()
2022-08-23T13:34:55.717214851Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.717220859Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.717226438Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.717232287Z self._read_from_socket()
2022-08-23T13:34:55.717246699Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.717259577Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.717579891Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.717602573Z return do()
2022-08-23T13:34:55.717608814Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.717614951Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.717620339Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.717626387Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.717631526Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.717636889Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.717656319Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.717663076Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.717668704Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.717687530Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.717693639Z Poller_0-21(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.717699344Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.717705932Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.717712088Z fail(error)
2022-08-23T13:34:55.717969038Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.717984571Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.717989162Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.717993281Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.717997520Z Traceback (most recent call last):
2022-08-23T13:34:55.718001600Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.718005908Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.718009727Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.718013864Z response = connection.read_response()
2022-08-23T13:34:55.718121752Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.718131617Z raw = self._buffer.readline()
2022-08-23T13:34:55.718621696Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.719176061Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.719227267Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.719235114Z response = connection.read_response()
2022-08-23T13:34:55.719241094Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.719247044Z return do()
2022-08-23T13:34:55.719252865Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.719259060Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.719264486Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.719270614Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.719275886Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.719282228Z raise error
2022-08-23T13:34:55.719288100Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.719293637Z self._read_from_socket()
2022-08-23T13:34:55.719310676Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.719317079Z self._read_from_socket()
2022-08-23T13:34:55.719322558Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.719328976Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.719372354Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.719379460Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.719385501Z File "/opt/librenms/LibreNMS/queuemanager.py", line 68, in _service_worker
2022-08-23T13:34:55.719392078Z device_id = self.get_queue(queue_id).get(True, 5)
2022-08-23T13:34:55.719397688Z File "/opt/librenms/LibreNMS/__init__.py", line 530, in get
2022-08-23T13:34:55.719402983Z item = self._redis.bzpopmin(self.key, timeout=timeout)
2022-08-23T13:34:55.719408126Z File "/usr/lib/python3.10/site-packages/redis/commands/core.py", line 4200, in bzpopmin
2022-08-23T13:34:55.719413732Z return self.execute_command("BZPOPMIN", *keys)
2022-08-23T13:34:55.719419422Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.719425376Z return do()
2022-08-23T13:34:55.719431142Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.719437564Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.719443053Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.719448621Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.719454646Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.719460617Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.719466348Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.719471671Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.719477765Z raw = self._buffer.readline()
2022-08-23T13:34:55.719483527Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.719489183Z self._read_from_socket()
2022-08-23T13:34:55.719494810Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.719500974Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.719506546Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.719535742Z raise error
2022-08-23T13:34:55.719541262Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1238, in execute_command
2022-08-23T13:34:55.719554669Z return conn.retry.call_with_retry(
2022-08-23T13:34:55.719560669Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 49, in call_with_retry
2022-08-23T13:34:55.719568657Z fail(error)
2022-08-23T13:34:55.719574149Z Discovery_0-8(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.719579541Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.719585761Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.719591807Z response = connection.read_response()
2022-08-23T13:34:55.719596877Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.719602196Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.719607380Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.719613455Z return do()
2022-08-23T13:34:55.719712428Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1242, in <lambda>
2022-08-23T13:34:55.719729037Z lambda error: self._disconnect_raise(conn, error),
2022-08-23T13:34:55.719735439Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.719741566Z raise error
2022-08-23T13:34:55.719970311Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.719991078Z return do()
2022-08-23T13:34:55.720206732Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.720222355Z raw = self._buffer.readline()
2022-08-23T13:34:55.720226997Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.720485766Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.720518021Z raw = self._buffer.readline()
2022-08-23T13:34:55.721082124Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.721102413Z self._read_from_socket()
2022-08-23T13:34:55.721219846Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.721265600Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.721275082Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.721281895Z raw = self._buffer.readline()
2022-08-23T13:34:55.721494602Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.721514376Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.721528130Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.721532457Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.721536262Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.721539944Z self._read_from_socket()
2022-08-23T13:34:55.721543401Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.721547385Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.721551000Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.721554707Z response = connection.read_response()
2022-08-23T13:34:55.721569902Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.721664060Z Discovery_0-15(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.721673076Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.722035706Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.722052537Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.722240359Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.722408912Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.722433993Z raw = self._buffer.readline()
2022-08-23T13:34:55.722537056Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.722589613Z self._read_from_socket()
2022-08-23T13:34:55.722599315Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.722603543Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.722607226Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.722611055Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.722614744Z self._read_from_socket()
2022-08-23T13:34:55.722685503Z Exception in thread Poller_0-19:
2022-08-23T13:34:55.722718946Z Exception in thread Services_0-6:
2022-08-23T13:34:55.722905963Z Traceback (most recent call last):
2022-08-23T13:34:55.722924279Z Services_0-7(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.722929580Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.722934056Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.722938637Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.722943057Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.722957595Z response = connection.read_response()
2022-08-23T13:34:55.723023634Z Exception in thread Poller_0-12:
2022-08-23T13:34:55.723044102Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.723048759Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.723052902Z Traceback (most recent call last):
2022-08-23T13:34:55.723214633Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.723239666Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.723325660Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.723354191Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.723360450Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1228, in _disconnect_raise
2022-08-23T13:34:55.723364865Z raise error
2022-08-23T13:34:55.723474480Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.723495615Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.723715113Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.723792785Z response = connection.read_response()
2022-08-23T13:34:55.723840029Z Traceback (most recent call last):
2022-08-23T13:34:55.723849395Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 611, in connect
2022-08-23T13:34:55.724063657Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.724080645Z raw = self._buffer.readline()
2022-08-23T13:34:55.724086677Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.724093174Z self._read_from_socket()
2022-08-23T13:34:55.724098730Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.724104586Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.724110014Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1254, in parse_response
2022-08-23T13:34:55.724115739Z response = connection.read_response()
2022-08-23T13:34:55.724122877Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 611, in connect
2022-08-23T13:34:55.724224432Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.724543697Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 611, in connect
2022-08-23T13:34:55.724567022Z Exception in thread Alerting:
2022-08-23T13:34:55.725261477Z Traceback (most recent call last):
2022-08-23T13:34:55.725593819Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.725613974Z raw = self._buffer.readline()
2022-08-23T13:34:55.725619032Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.725623366Z self._read_from_socket()
2022-08-23T13:34:55.725627284Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.725631374Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.725680049Z File "/usr/lib/python3.10/site-packages/redis/retry.py", line 46, in call_with_retry
2022-08-23T13:34:55.725703150Z return do()
2022-08-23T13:34:55.726147228Z sock = self.retry.call_with_retry(
2022-08-23T13:34:55.726172808Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 611, in connect
2022-08-23T13:34:55.726252789Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.726269911Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.726524284Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.727404938Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 318, in read_response
2022-08-23T13:34:55.727424133Z raw = self._buffer.readline()
2022-08-23T13:34:55.727428561Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.727432658Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1239, in <lambda>
2022-08-23T13:34:55.727436968Z lambda: self._send_command_parse_response(
2022-08-23T13:34:55.727968363Z File "/usr/lib/python3.10/site-packages/redis/client.py", line 1215, in _send_command_parse_response
2022-08-23T13:34:55.727987022Z return self.parse_response(conn, command_name, **options)
2022-08-23T13:34:55.727991596Z Poller_0-2(CRITICAL):BZPOPMIN/ZPOPMIN command failed: Connection closed by server.
2022-08-23T13:34:55.727995221Z Note that redis >= 5.0 is required.
2022-08-23T13:34:55.727999106Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 195, in _read_from_socket
2022-08-23T13:34:55.728002951Z raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
2022-08-23T13:34:55.728006566Z redis.exceptions.ConnectionError: Connection closed by server.
2022-08-23T13:34:55.728010592Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 824, in read_response
2022-08-23T13:34:55.728014449Z response = self._parser.read_response(disable_decoding=disable_decoding)
2022-08-23T13:34:55.728126417Z File "/usr/lib/python3.10/site-packages/redis/connection.py", line 249, in readline
2022-08-23T13:34:55.728143577Z self._read_from_socket()
....
This issue might be isolated to instances built in Docker. Our instance is built using traditional Linux VMs, and when we tested a failure of Redis and RRDCache, and polling paused while they were down, but began soon after they came back up.
