django-q2 icon indicating copy to clipboard operation
django-q2 copied to clipboard

Published 20 hours ago • verified publisher icon GDay

'qcluster' doesn't stop with Control+C

Open JoeHitchen opened this issue 2 years ago • 7 comments

Thank you for taking over maintaining this project! I'm really enjoying having an alternative to Celery to play with, and my end goal is a simple containerised setup where I have a worker container running django-q2 alongside a server container handling web requests.

I'm struggling a bit to develop with django-q though because of this issue, whereby the qcluster command appears to self-heal instead of gracefully exiting upon receiving a KeyboardInterrupt/Control+C message. This results in terminal windows which just need to be closed and set up again, and containers which keep restarting themselves even when told to stop.

My logs for such a cluster running without tasks look like:

python manage.py qcluster -v 3 20:21:26 [Q] INFO Q Cluster twelve-vermont-ack-winner starting. 20:21:27 [Q] INFO Process-1 guarding cluster twelve-vermont-ack-winner 20:21:27 [Q] INFO Q Cluster twelve-vermont-ack-winner running. 20:21:28 [Q] INFO Process-1:9 monitoring at 16640 20:21:28 [Q] INFO Process-1:1 ready for work at 18196 20:21:28 [Q] INFO Process-1:5 ready for work at 288 20:21:28 [Q] INFO Process-1:7 ready for work at 12476 20:21:28 [Q] INFO Process-1:4 ready for work at 9580 20:21:28 [Q] INFO Process-1:2 ready for work at 7452 20:21:28 [Q] INFO Process-1:6 ready for work at 3120 20:21:28 [Q] INFO Process-1:3 ready for work at 4484 20:21:28 [Q] INFO Process-1:10 pushing tasks at 9376 20:21:28 [Q] INFO Process-1:8 ready for work at 4828 KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt KeyboardInterrupt 20:25:00 [Q] ERROR reincarnated worker Process-1:5 after death 20:25:00 [Q] ERROR reincarnated pusher Process-1:10 after sudden death 20:25:00 [Q] ERROR reincarnated worker Process-1:1 after death 20:25:00 [Q] ERROR reincarnated worker Process-1:3 after death 20:25:00 [Q] ERROR reincarnated worker Process-1:6 after death 20:25:00 [Q] ERROR reincarnated worker Process-1:8 after death 20:25:00 [Q] ERROR reincarnated monitor Process-1:9 after sudden death 20:25:01 [Q] INFO Process-1:12 pushing tasks at 17868 20:25:01 [Q] INFO Process-1:11 ready for work at 2424 20:25:01 [Q] ERROR reincarnated worker Process-1:2 after death 20:25:01 [Q] ERROR reincarnated worker Process-1:7 after death 20:25:01 [Q] INFO Process-1:13 ready for work at 11524 20:25:01 [Q] INFO Process-1:14 ready for work at 1356 20:25:01 [Q] INFO Process-1:15 ready for work at 15292 20:25:01 [Q] INFO Process-1:17 monitoring at 6928 20:25:01 [Q] INFO Process-1:16 ready for work at 3184 20:25:01 [Q] ERROR reincarnated worker Process-1:4 after death 20:25:02 [Q] INFO Process-1:18 ready for work at 9392 20:25:02 [Q] INFO Process-1:19 ready for work at 5960 20:25:02 [Q] INFO Process-1:20 ready for work at 6156`

JoeHitchen avatar Jan 26 '23 20:01 JoeHitchen

Thanks for raising this issue here too!

I am failing to replicate the issue within my setup. How does your containerised setup look like? Have you also tried it with the django-q2 version?

I am using docker-compose and I haven't had any issues with a setup similar to this:

version: '3'

services:
  web:
    restart: always
    command: python manage.py runserver 0.0.0.0:8000
    ports:
      - "127.0.0.1:8000:8000"
    build: .
    volumes:
      - .:/app

  qcluster:
    restart: always
    command: python manage.py qcluster
    build: .
    volumes:
      - .:/app

GDay avatar Jan 27 '23 01:01 GDay

Well I just found out Ctrl + Break in Windows Terminal can stop qcluster. btw on my thinkpad notebook that's a Ctrl + Fn + B

sinowood avatar Jan 27 '23 03:01 sinowood

@sinowood - I can confirm that Ctrl + Fn + B works for me locally too, thank you!

It inserts the ^C character, which I guess is handled differently to the KeyboardInterrupt exception. It might be useful to have KeyboardInterrupt handled the same way, since that is how most people will be trying to stop it (you can see the activity on the original thread!) at first - certainly I did not know about Ctrl + Fn + B until just now - but at least there is a workaround!

JoeHitchen avatar Jan 27 '23 10:01 JoeHitchen

@GDay - Don't have my docker config in front of me right now and want to sanity check it again before sharing, but will try to get back to you soon - I'm pretty sure I wasn't doing anything radically different to your template though...

Would be happy to separate that into a new thread if you want to keep it distinct from the KeyboardInterrupt/Control+C discussion.

Appreciate the quick reply!

JoeHitchen avatar Jan 27 '23 10:01 JoeHitchen

Sanity check failed - The restarting containers were not caused by anything within Django-Q!

Happy for this to be closed, unless you are interested in keeping the suggestion of changing the KeyboardInterrupt behaviour.

JoeHitchen avatar Jan 27 '23 18:01 JoeHitchen

Sorry for the late reaction!

unless you are interested in keeping the suggestion of changing the KeyboardInterrupt behaviour.

Hmmm... I think KeyboardInterrupt is being caught, but the ctrl + c message isn't always correct, as it seems to vary per device what keys need to be pressed (on windows, I think). Windows does seem to handle these things differently.

I did find this: https://danielkaes.wordpress.com/2009/06/04/how-to-catch-kill-events-with-python/ I am not entirely sure if that would work, but it might be worth a shot.

GDay avatar Feb 05 '23 00:02 GDay

I can confirm the problem, I started a new Django project just to test it. There's nothing done in the project except configuration as shown in docs.

This is my settings.py file:

"""
Django settings for backend project.

Generated by 'django-admin startproject' using Django 5.0.2.

For more information on this file, see
https://docs.djangoproject.com/en/5.0/topics/settings/

For the full list of settings and their values, see
https://docs.djangoproject.com/en/5.0/ref/settings/
"""

from pathlib import Path

# Build paths inside the project like this: BASE_DIR / 'subdir'.
BASE_DIR = Path(__file__).resolve().parent.parent

# Quick-start development settings - unsuitable for production
# See https://docs.djangoproject.com/en/5.0/howto/deployment/checklist/

# SECURITY WARNING: keep the secret key used in production secret!
SECRET_KEY = 'django-insecure-j^!&h+okb*!*1st1(5ulp%@it+e0mhqs)yohmpf&dqqh+ya)98'

# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True

ALLOWED_HOSTS = []

# Application definition

INSTALLED_APPS = [
    'django.contrib.admin',
    'django.contrib.auth',
    'django.contrib.contenttypes',
    'django.contrib.sessions',
    'django.contrib.messages',
    'django.contrib.staticfiles',
    'django_q',
    'core',
]

MIDDLEWARE = [
    'django.middleware.security.SecurityMiddleware',
    'django.contrib.sessions.middleware.SessionMiddleware',
    'django.middleware.common.CommonMiddleware',
    'django.middleware.csrf.CsrfViewMiddleware',
    'django.contrib.auth.middleware.AuthenticationMiddleware',
    'django.contrib.messages.middleware.MessageMiddleware',
    'django.middleware.clickjacking.XFrameOptionsMiddleware',
]

ROOT_URLCONF = 'backend.urls'

TEMPLATES = [
    {
        'BACKEND': 'django.template.backends.django.DjangoTemplates',
        'DIRS': [],
        'APP_DIRS': True,
        'OPTIONS': {
            'context_processors': [
                'django.template.context_processors.debug',
                'django.template.context_processors.request',
                'django.contrib.auth.context_processors.auth',
                'django.contrib.messages.context_processors.messages',
            ],
        },
    },
]

WSGI_APPLICATION = 'backend.wsgi.application'

# Database
# https://docs.djangoproject.com/en/5.0/ref/settings/#databases

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.sqlite3',
        'NAME': BASE_DIR / 'db.sqlite3',
    }
}

# Password validation
# https://docs.djangoproject.com/en/5.0/ref/settings/#auth-password-validators

AUTH_PASSWORD_VALIDATORS = [
    {
        'NAME': 'django.contrib.auth.password_validation.UserAttributeSimilarityValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.MinimumLengthValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.CommonPasswordValidator',
    },
    {
        'NAME': 'django.contrib.auth.password_validation.NumericPasswordValidator',
    },
]

# Internationalization
# https://docs.djangoproject.com/en/5.0/topics/i18n/

LANGUAGE_CODE = 'en-us'

TIME_ZONE = 'UTC'

USE_I18N = True

USE_TZ = True

# Static files (CSS, JavaScript, Images)
# https://docs.djangoproject.com/en/5.0/howto/static-files/

STATIC_URL = 'static/'

# Default primary key field type
# https://docs.djangoproject.com/en/5.0/ref/settings/#default-auto-field

DEFAULT_AUTO_FIELD = 'django.db.models.BigAutoField'

# Django-Q2 - native Django task queue, scheduler and worker application
# https://django-q2.readthedocs.io/en/master/configure.html
Q_CLUSTER = {
    'name': 'DjangORM',
    'workers': 4,
    'timeout': 90,
    'retry': 120,
    'queue_limit': 50,
    'bulk': 10,
    'orm': 'default'
}

When I run qcluster, that's what's shown in CMD window:

(venv) F:\djng\queues\backend>python manage.py qcluster
17:35:15 [Q] INFO Q Cluster friend-pizza-equal-don starting.
17:35:16 [Q] INFO Process-a77363f5bc964465b5df74695b528501 guarding cluster friend-pizza-equal-don [DjangORM]
17:35:16 [Q] INFO Q Cluster friend-pizza-equal-don [DjangORM] running.
17:35:17 [Q] INFO Process-7745db6fe27e4b18b0561d42cda88855 ready for work at 32168
17:35:17 [Q] INFO Process-ef671c1d45754c5eab0187cf4604f866 ready for work at 19856
17:35:17 [Q] INFO Process-805149ca5ff14141a6e638a82eba5f88 ready for work at 27652
17:35:17 [Q] INFO Process-9092f476f014444bac154c109db36398 ready for work at 25336
17:35:17 [Q] INFO Process-bfb64627f9b745fdbfec1aed80369bdf monitoring at 20548
17:35:17 [Q] INFO Process-c11f21f884b049d4884e8ce9a2b7f139 pushing tasks at 40564

Then Ctrl+C is pressed and...

Process Process-805149ca5ff14141a6e638a82eba5f88:
Process Process-9092f476f014444bac154c109db36398:
Process Process-ef671c1d45754c5eab0187cf4604f866:
Process Process-7745db6fe27e4b18b0561d42cda88855:
Process Process-bfb64627f9b745fdbfec1aed80369bdf:
Process Process-c11f21f884b049d4884e8ce9a2b7f139:
Traceback (most recent call last):
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "F:\djng\queues\venv\Lib\site-packages\django_q\pusher.py", line 45, in pusher
    task_set = broker.dequeue()
               ^^^^^^^^^^^^^^^^
  File "F:\djng\queues\venv\Lib\site-packages\django_q\brokers\orm.py", line 85, in dequeue
    sleep(Conf.POLL)
Traceback (most recent call last):
Traceback (most recent call last):
KeyboardInterrupt
Traceback (most recent call last):
Traceback (most recent call last):
Traceback (most recent call last):
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "F:\djng\queues\venv\Lib\site-packages\django_q\worker.py", line 57, in worker
    for task in iter(task_queue.get, "STOP"):
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "F:\djng\queues\venv\Lib\site-packages\django_q\queues.py", line 72, in get
    x = super(Queue, self).get(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "F:\djng\queues\venv\Lib\site-packages\django_q\worker.py", line 57, in worker
    for task in iter(task_queue.get, "STOP"):
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 314, in _bootstrap
    self.run()
  File "C:\Program Files\Python312\Lib\multiprocessing\queues.py", line 102, in get
    with self._rlock:
  File "C:\Program Files\Python312\Lib\multiprocessing\synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "F:\djng\queues\venv\Lib\site-packages\django_q\queues.py", line 72, in get
    x = super(Queue, self).get(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "C:\Program Files\Python312\Lib\multiprocessing\process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
KeyboardInterrupt
  File "F:\djng\queues\venv\Lib\site-packages\django_q\worker.py", line 57, in worker
    for task in iter(task_queue.get, "STOP"):
  File "C:\Program Files\Python312\Lib\multiprocessing\queues.py", line 102, in get
    with self._rlock:
  File "F:\djng\queues\venv\Lib\site-packages\django_q\monitor.py", line 44, in monitor
    for task in iter(result_queue.get, "STOP"):
  File "F:\djng\queues\venv\Lib\site-packages\django_q\worker.py", line 57, in worker
    for task in iter(task_queue.get, "STOP"):
  File "F:\djng\queues\venv\Lib\site-packages\django_q\queues.py", line 72, in get
    x = super(Queue, self).get(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\djng\queues\venv\Lib\site-packages\django_q\queues.py", line 72, in get
    x = super(Queue, self).get(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "F:\djng\queues\venv\Lib\site-packages\django_q\queues.py", line 72, in get
    x = super(Queue, self).get(*args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\queues.py", line 103, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
  File "C:\Program Files\Python312\Lib\multiprocessing\queues.py", line 103, in get
    res = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\connection.py", line 216, in recv_bytes
    buf = self._recv_bytes(maxlength)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\queues.py", line 102, in get
    with self._rlock:
  File "C:\Program Files\Python312\Lib\multiprocessing\connection.py", line 321, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\connection.py", line 321, in _recv_bytes
    waitres = _winapi.WaitForMultipleObjects(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Program Files\Python312\Lib\multiprocessing\synchronize.py", line 95, in __enter__
    return self._semlock.__enter__()
           ^^^^^^^^^^^^^^^^^^^^^^^^^
KeyboardInterrupt
KeyboardInterrupt
KeyboardInterrupt
17:36:34 [Q] CRITICAL reincarnated worker Process-805149ca5ff14141a6e638a82eba5f88 after death
17:36:34 [Q] CRITICAL reincarnated worker Process-ef671c1d45754c5eab0187cf4604f866 after death
17:36:34 [Q] CRITICAL reincarnated monitor Process-bfb64627f9b745fdbfec1aed80369bdf after sudden death
17:36:34 [Q] CRITICAL reincarnated pusher Process-c11f21f884b049d4884e8ce9a2b7f139 after sudden death
17:36:35 [Q] CRITICAL reincarnated worker Process-9092f476f014444bac154c109db36398 after death
17:36:35 [Q] INFO Process-f30c22cc7e2649c5867b070d69ec9b05 ready for work at 39560
17:36:35 [Q] INFO Process-8b13b2c5d4e84860b6c360924fb32fc5 monitoring at 17108
17:36:35 [Q] INFO Process-77777145be2f4518b81b9ead0f36933c ready for work at 10836
17:36:35 [Q] CRITICAL reincarnated worker Process-7745db6fe27e4b18b0561d42cda88855 after death
17:36:35 [Q] INFO Process-b94f0e1cb8bc47d4a6a4f920293cfeb4 pushing tasks at 32472
17:36:36 [Q] INFO Process-fcfe42969243432ba80c6e8806be0fb5 ready for work at 27488
17:36:36 [Q] INFO Process-1eca3d77fa4f4c6b8a54c65a8b7aa463 ready for work at 6584

VBobCat avatar Feb 29 '24 20:02 VBobCat