Implement Redis backend

[epwalsh]

[morenol]

I would like to take this, do you have any thoughts, suggestions, guideline that I should consider before jumping into it?

[epwalsh]

Hey @morenol! That would be great. I think the steps to tackle this are as follows:

1. Start by designing a Backend trait, and show how the Celery app would interact with it. You could submit this as a PR or even just post a code-block in this thread with some explanation.
2. Figure out how Python's Celery implements the Redis backend under the hood. In particular, we need to know the serialization format and which Redis functionality is used so that we can be compatible with the Python implementation. This might be the hardest part since the internals of Python Celery are not well documented.
3. Implement the RedisBackend based on what we've found in Step 2.
4. Add unit tests and integration tests.
5. Bonus points: add an integration test that checks cross compatibility with the Python implementation.

[morenol]

That sounds like a good plan!

[0x0c72]

I have been working on this somewhat as I would personally find it useful. AsyncResult still needs altering to work with backends, AsyncBackend and ResultConsumer are also not finished. The parallel to Message/Delivery for backends still needs to be settled. And lastly, none of the actual hooking up of backends in the main app has been finished nor have tests been written.

Despite the work remaining, a code review would be very helpful at this point as well as some architectural discussion. If anyone is willing to engage I will drop a PR. As of now it compiles and all of the previous functionality works despite any changes. Test cases are also giving same results as before.

If there is interest, I will compose a more detailed write up of the work alongside the PR. Otherwise I will just keep hacking along on my own. 👍

[PiDelport]

Hi folks! For anyone interested, I implemented a very quick and dirty persistence backend for the project I'm working on. The code is under this task_records module:

• types – The TaskRecord type, which records a task execution result.
• redis_impl – RedisTaskRecords, which implements saving and retrieving TaskRecord values, and provides a helper to send tasks and retrieve their results as one action.
• callbacks – Task on_success / on_failure callbacks to save results using RedisTaskRecords.

This implementation only covers our project's most immediate needs, so it has a lot of limitations and cautions, but it might provide a useful reference point for further work on this issue.

[haxorcize]

This PR is awesome. Any way we can complete the review and merge it?