Switching 'ASYNC' Configuration Dynamically In Testing

[MicahLyle]

I'd like the ability to dynamically switch during tests whether queued calls, using django_rq.enqueue or function_name.delay(...) by default run with is_async True or False if not specified by the enqueue or delay call. I wrote a context manager (meant to be used in an instance of Django's TestCase) below, and I don't understand why it's not working. Would love some input:

@contextmanager
def rq_jobs_async(self, yes_or_no: bool):
    """
    If `yes_or_no` is `True`, jobs execute asynchronously and need to be run by workers.
    If `yes_or_no` is `False`, jobs execute synchronously and don't need to be run by workers.
    """
    QUEUES = django_rq.settings.QUEUES
    originals = {}
    # https://github.com/rq/django-rq#synchronous-mode
    for name, queue_config in settings.RQ_QUEUES.items():
        # NOTE: We change both the settings and `queue`. Realistically,
        # you only need to change `queue` since the settings are only
        # used in the beginning to set `_is_async` on the queue.
        # However, keeping the settings here in case that ever changes in the
        # future and/or other code accesses the settings for whatever reason.
        originals[name] = queue_config['ASYNC']
        queue_config['ASYNC'] = yes_or_no
        QUEUES[name]['ASYNC'] = yes_or_no
        queue = django_rq.get_queue(name)
        queue._is_async = yes_or_no
    try:
        yield
    finally:
        # Restore the `RQ_QUEUES` 'ASYNC' values to their originals,
        # the 'QUEUES' 'ASYNC' values to their originals, and
        # the `_is_async` property of each `queue` to its original.
        for name, queue_config in settings.RQ_QUEUES.items():
            queue_config['ASYNC'] = originals[name]
            QUEUES[name]['ASYNC'] = originals[name]
            queue = django_rq.get_queue(name)
            queue._is_async = originals[name]

It seems that no matter how I try and do it, the is_async setting (whether Django's setting or django-rq's setting from its specific file or the actual _is_async attribute of the Queue) ends up being whatever the settings originally set it to. So if the initial settings says 'ASYNC' is False, then django-rq seems to under the hood will treat 'ASYNC' as False, even if overriden in its own settings, Django settings, or on a queue level.

Has anyone managed to switch is_async or queue_config['ASYNC'] dynamically in tests or monkeypatch them and have things work?

[selwin]

I think this is because variables in https://github.com/rq/django-rq/blob/master/django_rq/settings.py is generated when your project is first run and is cached.

I think it'd be easier to add this line to Django's settings.py so you don't need to modify your app logic:

if 'test' in sys.argv:
    RQ_QUEUES = {
        'default': {
            'HOST': 'localhost',
            'PORT': 6379,
            'DB': 0,
            'ASYNC': False,
        },
    }

Alternatively, I'm also open to suggestions on reworking django-rq.settings.py to parse RQ_QUEUES during runtime so the behavior can be changed via Django's override_settings.

[MicahLyle]

Thanks! For now I've just resorted to manually running workers when I need to emulate what it would be like to run an async worker.

[striveforbest]

Seems like the jobs called with .delay() still fail if Redis is not running despite having ASYNC set to False. Here are my settings:

Out[1]:
{'default': {'HOST': 'localhost',
  'PORT': 6379,
  'DB': 0,
  'PASSWORD': '',
  'DEFAULT_TIMEOUT': 600,
  'ASYNC': False},
 'scheduled': {'HOST': 'localhost',
  'PORT': 6379,
  'DB': 0,
  'PASSWORD': '',
  'DEFAULT_TIMEOUT': 7200,
  'ASYNC': False}}

The jobs fail because there is nothing running on port 6379. Is there a setting that is similar to Celery's ALWAYS_EAGER?

[selwin]

@StriveForBest setting ASYNC to False will run jobs synchronously, but you'll still need Redis, because it will store job information and results in Redis.

[striveforbest]

@selwin yea i figured that much. I have a solution but it feels hacky and depends on another library. I wish there was just a flag. But in case someone is interested, in my settings.py:

from fakeredis import FakeStrictRedis, FakeRedis

RQ_REDIS_ENABLED = False

if not RQ_REDIS_ENABLED:
    import django_rq.queues
    django_rq.queues.get_redis_connection = lambda _, strict: FakeStrictRedis() if strict else FakeRedis()

[BobReid]

I just ran into this. It used to work fine for me but does not work after switching from an explicit enqueue call to the job decorator.

I believe the problem is the django_rq vesion of the decorator passes an actual queue object into the base version of the decorator and not just the queue name. This means that all the configuration of the queue is read at initialization and not when delay is actually called.

I think the django_rq decororator should pass the queue name to preserve this lazy loading of the configuration but I suspect the base decorator will not read the queue configuration from the django conf.

[selwin]

Mind opening a PR for this?

On Mar 4, 2021, at 4:17 AM, BobReid [email protected] wrote:

 I just ran into this. It used to work fine for me but does not work after switching from an explicit enqueue call to the job decorator.

I believe the problem is the django_rq vesion of the decorator passes an actual queue object into the base version of the decorator and not just the queue name. This means that all the configuration of the queue is read at initialization and not when delay is actually called.

I think the django_rq decororator should pass the queue name to preserve this lazy loading of the configuration but I suspect the base decorator will not read the queue configuration from the django conf.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or unsubscribe.