rusty-celery
rusty-celery copied to clipboard
rusty-celery
Creating tasks from inside running task
Is your feature request related to a problem? Please describe. Ability to create tasks from other tasks. Currently, as I understand, the only way to publish tasks is via Celery app. I would like to have ability to create one tasks from already running task.
Additional context I have a use case, where I would like to use Rust and Celery for distributing tasks. In this case I need to have a Rust task, which works with some data, splits it and passes it to separate tasks (could be up to 1k of separate small tasks), which are then processed separately.
Hey @retrry, this is definitely an important feature to have. I'm going to put it 0.4.0 milestone so that this becomes a priority. Right now I think the best way to implement this would be to add a reference to the Celery app within the Request struct. That way you could write your task like this:
#[celery::task(bind = true)]
async fn create_more_tasks(task: &Self) {
task.request.app.send_task(other_task::new()).await?;
}
I can work on this later this week unless someone else wants to tackle it.
@yageek are interested in trying to implement my proposed solution above? I can try to provide more guidance if you need it.
I would be interested on more insights. I tried some changes but I did not succeed.
The difficulty I had was to carry the generic 'Broker' element from the 'Celery' struct into the 'Request' type and the 'Message' type along the way.
Hmm maybe we need to make a trait (let's call it CeleryBase or something) that represents a celery app (the Celery struct would impl this trait). This trait would include the send_task method of course. Then the Request struct could hold a Box<&dyn CeleryBase>.
Glas to hear it 😛 I tried using a trait also for the Celery (actually named it Dispatcher) and failed to implement it using static dispatch. I'll give another try using dynamic dispatch/trait object.
@epwalsh do you have some tips about how to deal with trait object and the generics used inside the send_task method?
@yageek are you running into a specific compiler error? If you've already started working on a branch, could you open up a draft PR so I can take a look?
@epwalsh I made several attempts but I can not succeed to deal with either dynamic/static dispatch :(
Using aBox<&dyn CeleryBase>. in Request does not work because of the generics (See Object safety
I'm trying another approach implying spreading around the Dispatcher type but this require me at some point to create self-referential struct and I am too noob with Pin.
@epwalsh is there a working workaround for this nowadays? 0.4.0 was released but this issue is still open so i'm guessing not?
