Wait asynchronously for a deferred job to complete and get its return value

[Korijn]

Question is in the title. With this library, can you (asynchronously) wait for a deferred job to complete and get its return value? I couldn't find it in the docs.

Something like:

result = await sum.defer_async(a=3, b=5)
output = await result.get()

[elemoine]

No, there's no such thing. The return values of tasks are ignored by Procrastinate. They are actually logged, but that's it. In particular they're not stored in the procrastinate_jobs table. So it is your task's responsibility to save its results somewhere, in an application-specific Postgres table for example.

Note that Task.defer returns a the deferred job id, that is the id of the job in the procrastinate_jobs table. Procrastinate doesn't currently provide a Python function for checking the status of a job given its id, but this is something we could consider adding in the future. In the mean time, you can use that job id to yourself read the job status from the procrastinate_jobs table.

[Korijn]

So, let me see if I understand correctly:

• I can get a job's status from the table procrastinate_jobs, but I would need to write the query myself
• Waiting for a job to complete is also something I would need to write myself? (I thought I read something about postgres' LISTEN being used...)
• Writing results and retrieving them should be implemented in application code

[elemoine]

* I can get a job's  status from the table `procrastinate_jobs`, but I would need to write the query myself

Yep. Contrary to what I said in my previous comment, Procrastinate does have a function for retrieving a job by its id, but it's not part of the official API yet.

jobs = app.manager.list_jobs(id=your_job_id)
job = jobs[0]
print(f"{job.status}")

But this is an area of Procrastinate that is still in flux I'd say.

Waiting for a job to complete is also something I would need to write myself?

Yes.

I thought I read something about postgres' LISTEN being used...

Yes, it's used internally to wake up the Procrastinate workers when a new job is inserted into the queue.

Writing results and retrieving them should be implemented in application code

Yep, you got that right.

[Korijn]

Alright, thanks for confirming! I'm contemplating a move from Celery to procrastinate because it fits our setup so well (the locking features in particular), but having to implement something similar to a result backend ourselves might be too risky.

Final question: do you support anything similar to Celery's task chains, where one task's output is fed to the input of the next task in a sequence?

[elemoine]

Alright, thanks for confirming! I'm contemplating a move from Celery to procrastinate because it fits our setup so well (the locking features in particular), but having to implement something similar to a result backend ourselves might be too risky.

In our applications, we use a Postgres table for storing results. We have a Postgres database which contains both the Procrastinate and application-specific objects.

[ewjoachim]

Final question: do you support anything similar to Celery's task chains, where one task's output is fed to the input of the next task in a sequence?

Well, given that we do not store the return value of a job (...yet), that's not something we have. But using the lock, you can make the equivalent of a sequence and ensure that a group of tasks will run in order (defer the tasks with the same lock value in order, and they will run in order).

I think we could imagine a lot of things from what you're mentionning:

• Storing the results of a task (would probably limit to JSON, same as the task args)
• Adding a function to fetch a task in the official public API
• Have each task emit a postgres NOTIFY that we could listen for
• Have a method for awaiting a task completion (maybe with a timeout, or leave that to the caller)
• Provide primitives such as chains and chords (I don't know if we could make chords today) to ease migration from Celery

This all requires some time to setup, so if we agree, we could make scoped tickets. Would you be interested to participate in the development of some of those, @Korijn ?

[Korijn]

I would enjoy working on something like that, however I am currently plowing through a list of personal projects already so it will take some time before I would be able to get to it. Is that a problem?

[ewjoachim]

From my point of view, it's fine if it's not done soon, and I'll be glad to provide some help or guidance :) But if people want to tackle some steps before, it's first come first serve :) (as long as we create tickets and, ideally, assign ourselves to them on ping in them)

[ewjoachim]

Hm, a few months apart, I realize I gave a different answer regarding storing tasks. https://github.com/peopledoc/procrastinate/issues/419

@elemoine What's your opinion on that ?