dramatiq_dashboard icon indicating copy to clipboard operation
dramatiq_dashboard copied to clipboard

strange behavior; CPU100%, RAM100%

Open vkatochka opened this issue 5 years ago • 0 comments

Dramatiq 1.9.0
Redis server v=5.0.7, redis-cli 5.0.7
Ubuntu 18.04.4 LTS
Python 3.6.9

After requesting the URL http://domain.name/ddb/, the CPU consumption increases to 100%, the size of used RAM starts to grow gradually and reaches 100%, after which Redis falls.

ddb.py:

import bjoern
import dramatiq
from dramatiq.brokers.redis import RedisBroker
from dramatiq_dashboard import DashboardApp

broker = RedisBroker(host='10.0.0.5', port=6379, db=12)
broker.declare_queue('default')
dramatiq.set_broker(broker)
app = DashboardApp(broker=broker, prefix='/ddb')
bjoern.run(app, '127.0.0.1', 8081)

nginx:

location ~ '^/ddb/(.*)$' {
	proxy_pass http://127.0.0.1:8081/$1;
}

redis-cli:

10.0.0.5:6379[12]> KEYS *
(error) BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.

Redis log:

22662:M 16 Jun 2020 07:40:55.725 # Lua slow script detected: still in execution after 5006 milliseconds. You can try killing the script using the SCRIPT KILL command.

Dramatiq log:

[2020-06-16 07:37:26,654] [PID 5847] [Thread-3] [dramatiq.worker.ConsumerThread(default.DQ)] [INFO] Restarting consumer in 3.00 seconds.
[2020-06-16 07:37:26,667] [PID 5842] [Thread-2] [dramatiq.worker.ConsumerThread(default)] [CRITICAL] Consumer encountered an unexpected error.
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dramatiq/brokers/redis.py", line 340, in __next__
    data = self.message_cache.pop(0)
IndexError: pop from empty list

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dramatiq/worker.py", line 253, in run
    for message in self.consumer:
  File "/usr/local/lib/python3.6/dist-packages/dramatiq/brokers/redis.py", line 354, in __next__
    self.prefetch - self.outstanding_message_count,
  File "/usr/local/lib/python3.6/dist-packages/dramatiq/brokers/redis.py", line 275, in do_dispatch
    return dispatch(args=args, keys=keys)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 4073, in __call__
    return client.evalsha(self.sha, len(keys), *args)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3143, in evalsha
    return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ResponseError: BUSY Redis is busy running a script. You can only call SCRIPT KILL or SHUTDOWN NOSAVE.

After redis shutdown (over RAM 100%) ddb.py says:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/http.py", line 110, in __call__
    return path_handler(request, start_response, **match.groupdict())
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/http.py", line 115, in wrapper
    response = make_response(fn(self, request, *args, **kwargs))
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/http.py", line 124, in wrapper
    response = fn(self, request, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/application.py", line 79, in dashboard
    "workers": self.iface.workers
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/interface.py", line 77, in workers
    for name, timestamp, jobs_in_flight in self.do_get_workers():
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/interface.py", line 118, in do_dispatch
    return dispatch(args=[command, *args], keys=keys)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 4073, in __call__
    return client.evalsha(self.sha, len(keys), *args)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3143, in evalsha
    return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 739, in read_response
    response = self._parser.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 324, in read_response
    raw = self._buffer.readline()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 256, in readline
    self._read_from_socket()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 201, in _read_from_socket
    raise ConnectionError(SERVER_CLOSED_CONNECTION_ERROR)
redis.exceptions.ConnectionError: Connection closed by server.

If call a command "SCRIPT KILL" from the redis-cli

redis-cli:

10.0.0.5:6379[12]> SCRIPT KILL
OK

redis log:

22662:M 16 Jun 2020 07:42:17.334 # Lua script killed by user with SCRIPT KILL.

ddb.py says:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/http.py", line 110, in __call__
    return path_handler(request, start_response, **match.groupdict())
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/http.py", line 115, in wrapper
    response = make_response(fn(self, request, *args, **kwargs))
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/http.py", line 124, in wrapper
    response = fn(self, request, *args, **kwargs)
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/application.py", line 79, in dashboard
    "workers": self.iface.workers
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/interface.py", line 77, in workers
    for name, timestamp, jobs_in_flight in self.do_get_workers():
  File "/usr/local/lib/python3.6/dist-packages/dramatiq_dashboard/interface.py", line 118, in do_dispatch
    return dispatch(args=[command, *args], keys=keys)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 4073, in __call__
    return client.evalsha(self.sha, len(keys), *args)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 3143, in evalsha
    return self.execute_command('EVALSHA', sha, numkeys, *keys_and_args)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 901, in execute_command
    return self.parse_response(conn, command_name, **options)
  File "/usr/local/lib/python3.6/dist-packages/redis/client.py", line 915, in parse_response
    response = connection.read_response()
  File "/usr/local/lib/python3.6/dist-packages/redis/connection.py", line 756, in read_response
    raise response
redis.exceptions.ResponseError: Error running script (call to f_d0115eef4d1849468031df2d9d7beb468e019680): @user_script:72: Script killed by user with SCRIPT KILL...

vkatochka avatar Jun 16 '20 08:06 vkatochka