Is using lazy_loader a good idea - integrating dramatiq into django, flask, cement and others

[TomFreudenberg]

Issues

Working on an app while using Cement framework, I need the option to lazy bind the actors after the broker setup is done.

What OS are you using?

macOS Ventura

What version of Dramatiq are you using?

v1.15.0 (from pip install)

What did you do?

Example layout

- controller.py
+ actions /
  - __init__.py
  - tasks.py

tasks.py

import dramatiq
import requests

@dramatiq.actor(queue_name='count_words')
def count_words(url):
    response = requests.get(url)
    count = len(response.text.split(" "))
    print(f"There are {count} words at {url!r}.")

When using this in a Cement controller, the code will fail, while the broker is not setup when module gets imported.

controller.py

from cement import Controller, ex
import .actions as actions

class Demo(Controller):

    def example(self):
        actions.tasks.count_words.send('https://github.com')

That will fail, because the app hook to initialze the broker was not run when module actions was imported and actors had applied.

What did you expect would happen?

Lazy load of the actors so that the broker setup can run before

def register_rmq(app):
    rabbitmq_broker = RabbitmqBroker(url="amqp://user:test@localhost:5672/")
    dramatiq.set_broker(rabbitmq_broker)

How to solve?

I found a solution in using the package scientific-python/lazy_loader:

Apply lazing loading to the actions module does solve the whole problem.

actions/init.py

import lazy_loader as lazy

__getattr__, __dir__, __all__ = lazy.attach(
    __name__,
    submodules=['tasks']
)

With just that piece of code in the __init__.py of the actions package, the module is loaded only on first access of any function.

Here, when using:

actions.tasks.count_words.send('https://github.com')

At that time also the broker is initialized.

How to continue?

Wouldn't that be an good solution for dramatiq_django and dramatiq_flask as well?

Or, is there any parts I missed why this is not a good solution?

[TomFreudenberg]

I have prepared a running starter at: https://github.com/TomFreudenberg/cedra.

Thanks for any comments.

[nphilipp]

I have the same issue (with version 1.16.0, on Fedora Linux 40 FWIW) – having to declare the broker at the time the module is imported makes flexible configuration unnecessarily difficult.

@TomFreudenberg thanks for the workaround! I’ll use it for the time being, but would greatly appreciate if this (or another) method were documented for this use case.

[Bogdanp]

This looks like a cool idea, but I'm not sure what is actionable about this issue. I'd happily accept a PR adding this method to the cookbook!