add celery support

[ayr-ton]

Let's discuss how to add celery support, broke down the steps for starting working in PRs, and if still makes sense or Dramatiq/Huey instead.

[madkote]

Let's discuss how to add celery support, broke down the steps for starting working in PRs, and if still makes sense or Dramatiq/Huey instead.

Hi @ayr-ton , celery is in the plan - but I have not used that extensively -> a use case would be very helpful.

Regarding Dramatiq/Huey:

• Celery can be setup with Redis (a big plus for commercial use, whereby RabbitMQ is not always free to use)
• Dramatiq currently relies on RabbitMQ and REDIS
• Huey seems to be a good alternative also for commercial use.

We might start by any most popular framework, and that is Celery.

Any samples and use cases are welcome! From my side, I will refresh my knowledge with celery and try Dramatiq and Huey.

thanks for interest!

[ayr-ton]

@madkote I'm considering how to refactor my current case for a sample here. My worker process is not using the FastAPI main classes yet. This is the part I'm improving on it, and then I will be able to share some ideas. But I also chose Celery for its implementation. Your last message left me less concerned about that decision. All the cool kids asked me why not Dramatiq or Huey, because Celery this or that.

[ayr-ton]

I'm testing Rocketry for a simple scheduler, and I think this could be an easier path for first implementations, as it's more modern and close to FastAPI ways of thinking: https://github.com/Miksus/rocketry/discussions/78#discussioncomment-3398190 Of course, I don't think Rocketry can do the same thing as Celery (yet), but the code is scaling better and doesn't need magic connections/decorators for sharing context with FastAPI. The way of writing pipelines, calling existing functions, and moving returns around is very lightweight as it remembers FastApi ways of dependency injection. Also, it might not work for big pipelines and very demanding celery workloads replacements, but it would give you a good amount of power until you need a broker for tasks. After a few refactories of celery tasks and webhooks with FastAPI I still think they look like kinda ugly or unnecessarily verbose. Finishing my first PR draft with it Rocketry; I know we agreed on Celery, but if you think it doesn't fit I can still send a different draft with Celery later so you can compare the implementations and decide which ones are more close to plugins architecture. This was good research.