django-q2
django-q2 copied to clipboard
GDay
Allow specifying "daemonize" at a task level, rather than a cluster level
Maybe my understanding here is too weak and this isn't feasible, but I'd like to only mark some tasks to be non-daemonized.
In short, I have tasks that kick off Selenium processes. These fail with the error AssertionError: daemonic processes are not allowed to have children.
I can set daemonize_workers to False, but this applies to everything going forward. Is there any way to isolate this to just the tasks that need it?
If it's a worker-level concept, I can see why this would be hard and less appealing to implement.
Workers are spawn with or without a deamon based on the configuration: https://github.com/django-q2/django-q2/blob/88430c90147c07d48dd842759e4992593cf706d4/django_q/cluster.py#L181-L192
What you could do is spin up two clusters with different configurations using multiple queues: https://django-q2.readthedocs.io/en/master/cluster.html#multiple-queues. For each task you can specify which cluster to use. You can create one cluster with deamons and the other without.
Maybe that works for you?
@GDay , yup conceptually that should work. The defect it seems to me is extra overhead, as running two clusters just to have different configuration will be wasteful of system resources and less flexible to load shift between the two queues? E.g. I'd have to guess in advance how many workers to put on each.
Conceptually though, it does seem like specifying this at a task level would be possible, which is helpful to know.
Yeah, agreed. I wouldn't know any other way right now though, without making big changes to the code. The workers are assigned to tasks without having any context on the task itself, so if some workers would run without a deamon, it wouldn't be able to tell if it should put it back in the queue or run it itself. They just wait for the queue to get filled and then pick up the first task at hand. Whatever worker gets it first, runs it.
I guess the important distinction here is that daemonize is a property of the worker, not the task. I think you're right that the solution here is to two queues and therefore two clusters.