celery

celery copied to clipboard

Kubernetes is a very common deployment target these days. The most common way to deploy an application to Kubernetes is using the Helm package manager. We need to create a public chart and contribute it to the main charts repository.

See https://github.com/celery/celery/issues/4079#issuecomment-323267288 regarding implementing both the livenessProbe and the readinessProbe.

+1 for a template for such chart.

Was there ever any progress on this?

Not yet. I will try to do this in the next version.

For information, I did an airflow chart based on celery, but I think there could be much more in a celery chart (multi queue, autoscaler,...)

For what it's worth, a lot of us that use k8s do not like helm.

Getting a nice docker image for celery would be great and help a bunch of different use cases.

Helm is basically some yaml that would be glued on TOP of the docker image, and IMHO is not that valuable. I suspect helm will slowly die out as people move away, unless perhaps there is a big revamp that fixes all the "bad" parts of helm.

The big upside of helm is the community. You need a rabbitMQ production ready in your cluster? Helm install. You need to embeded rabbit + reddis + ... in your own chart ? => helm dependency build. In short you do not reimplement everything, you reuse a lot what already exists in your own chart.

Our Airflow package embeds celery, but also reuse reddis chart for example, so that I do not have to reimplement the deployment. Same for postgresQL. And if people wants to use their own instance (DBaaS) they could without having to rewrite the deployment.

That's very underevaluated. This is what makes helm the defacto standard for the moment, not the template things others do, in a mucher simpler way (kustomize, ksonnet,...). In short helm is:

• template (easy to have)
• tiller (much disliked, but helps a lot continuous deployement)
• package reuse
• good community and lot of good practices docs to write portable deployment recipes (charts).

Worth a try.

The problem with Helm, is the 75% problem. Often a chart solves 75% of your problem. Missing some flags that you need? Make a PR. Some new issue with a chart? Make a PR or do a local hack. It ends up taking much much more time to "fix" a chart, compared to just creating your own k8s yamls.

This is a much bigger topic than this issue, so perhaps not the right forum for it. But google around and you will see a lot of people that dipped toes in Helm have left, it does not seem to much a lot of momentum going forward.

Either way - To use helm you need a good docker image. For me to create my own deployment yaml, I need a good docker image. So the next step in either case is to create good docker images ;) If people want to ALSO create a helm chart - more power to them (but I won't use it). But I might indeed consider a docker image if made well.

I agree. Making a docker image is not difficult (we do have celery in several docker images here). BUT making a generic one is really tricky indeed.

:wave: Helm user here. I must say that having a helm chart ease a lot the deployment (I just hit this issue while trying to package a big app that use celery). I would have loved to see an actively maintained chart with production ready examples and real life feedbacks ^^"

I might draft one in the following days and, if it doesn't get too uggly, I might open source it if you don't might (I would be happy to share ownership as I have no experience whatsoever with celery). Also, if you consider publishing a chart, I highly suggest hosting your own repo and publish your chart on https://hub.helm.sh/ (So you don't need to go through all the validation process of publishing under the helm/kubernetes organization on github).

Edit: Don't get me wrong: I'm also not very happy with the state of helm as it is now, like many helm users but the draft for helm v3 dropping tiller and moving to an operator pattern will hopefully make everyone happy :D

As a maintainer for Helm Charts like redis, rabbitmq, etc, and heavy user of celery deployed using helm (currently ~100 celery pods running in prod), here are my ~~two~~ five cents:

1/ A good docker image is a must-have and comes before a good helm chart. A good Docker image does not necessarily means complex; it can be a few lines long. 2/ People are using more and more helm, in fact it seems like it is becoming the de-facto standard to deploy things in kube 3/ Helm itself is far from perfect and writing templates of yaml using even-weirder-than-jinja2-language can be a pain in the ass for complex things, but it should improve with helm 3. 4/ Helm charts are currently centralized in charts.git but are going to be decentralized soon (link already given here) 5/ I would be happy to review a celery helm chart either in charts.git or in a celery-controlled repository.

Edit: after some discussion with some member here, I realize that to use celery, one has to package its code onto the docker image, so one needs to rebuild for each project/commit the docker image. In this case, maybe all that Celery can do is create a base "abstract" image to be "inherited" by custom-celery-image-with-code that would just do some pip install. Or maybe there is no possible generic "celery" Docker at all...

Is there any progress/update on this?

I was wondering if Celery would like to see the adopt operator pattern in addition to the Helm chart. Let me know if I should create a separate issue for the same.

I built a very much POC(https://github.com/brainbreaker/Celery-Kubernetes-Operator) of Celery operator for solving the manual steps needed to manage celery clusters at my work.

There's also a discussion going on at https://github.com/celery/ceps/issues/24 for the same. I'm willing to commit a certain number of hours every week to build a production-ready version of the Operator. But I'd need inputs from the community around what will be the requirements, production use cases/opportunities that the operator needs to solve.

I also presented this operator POC in my recent talk at EuroPython 2020(Slides link - https://bit.ly/europython20-ppt) around automating the management of Kubernetes infra while staying in the Python ecosystem.

Is there any progress/update on this?

I was wondering if Celery would like to see the adopt operator pattern in addition to the Helm chart. Let me know if I should create a separate issue for the same.

I built a very much POC(https://github.com/brainbreaker/Celery-Kubernetes-Operator) of Celery operator for solving the manual steps needed to manage celery clusters at my work.

There's also a discussion going on at celery/ceps#24 for the same. I'm willing to commit a certain number of hours every week to build a production-ready version of the Operator. But I'd need inputs from the community around what will be the requirements, production use cases/opportunities that the operator needs to solve.

I also presented this operator POC in my recent talk at EuroPython 2020(Slides link - https://bit.ly/europython20-ppt) around automating the management of Kubernetes infra while staying in the Python ecosystem.

Yes, we're planning to replace celery multi which is very primitive with a concept of a controller. The purpose of the controller is to be able to manage the deployment on multiple platforms. You can read more about it in celery/ceps#27.

@brainbreaker If you have experience with Celery and you need that kind of thing at work I suggest you help us. We're working on the next generation of Celery.

Is there any progress/update on this?

I was wondering if Celery would like to see the adopt operator pattern in addition to the Helm chart. Let me know if I should create a separate issue for the same.

I built a very much POC(https://github.com/brainbreaker/Celery-Kubernetes-Operator) of Celery operator for solving the manual steps needed to manage celery clusters at my work.

There's also a discussion going on at celery/ceps#24 for the same. I'm willing to commit a certain number of hours every week to build a production-ready version of the Operator. But I'd need inputs from the community around what will be the requirements, production use cases/opportunities that the operator needs to solve.

I also presented this operator POC in my recent talk at EuroPython 2020(Slides link - https://bit.ly/europython20-ppt) around automating the management of Kubernetes infra while staying in the Python ecosystem.

I am checking your work. feel free to ping if you want me to collaborate on that effort

@brainbreaker If you have experience with Celery and you need that kind of thing at work I suggest you help us. We're working on the next generation of Celery.

I'd love to. I'm not an expert with Celery internals right now but willing to explore and learn. We use it at our work for various asynchronous workloads and there are pain points of managing all the workers, flower, and scaling things across multiple celery use cases.

I am checking your work. feel free to ping if you want me to collaborate on that effort

Thanks, @auvipy. Code is certainly not the best quality right now so please bear with that. I'll definitely need your inputs.

@thedrow Also, please correct me if I'm understanding wrong - The Controller concept we're introducing in Celery 5 will be platform agnostic, and then we'll provide packages(Platform manager), one of which could be the Kubernetes Operator which will be using that controller internally.

@brainbreaker If you have experience with Celery and you need that kind of thing at work I suggest you help us. We're working on the next generation of Celery.

I'd love to. I'm not an expert with Celery internals right now but willing to explore and learn. We use it at our work for various asynchronous workloads and there are pain points of managing all the workers, flower, and scaling things across multiple celery use cases.

I am checking your work. feel free to ping if you want me to collaborate on that effort

Thanks, @auvipy. Code is certainly not the best quality right now so please bear with that. I'll definitely need your inputs.

@thedrow Also, please correct me if I'm understanding wrong - The Controller concept we're introducing in Celery 5 will be platform agnostic, and then we'll provide packages(Platform manager), one of which could be the Kubernetes Operator which will be using that controller internally.

you can currently start with celery 4.4.x and later when the celery 5 enters in rc1 stage we can move the effort to that. (I expect 6-8 months more around)

@brainbreaker If you have experience with Celery and you need that kind of thing at work I suggest you help us. We're working on the next generation of Celery.

I'd love to. I'm not an expert with Celery internals right now but willing to explore and learn. We use it at our work for various asynchronous workloads and there are pain points of managing all the workers, flower, and scaling things across multiple celery use cases.

I am checking your work. feel free to ping if you want me to collaborate on that effort

Thanks, @auvipy. Code is certainly not the best quality right now so please bear with that. I'll definitely need your inputs. @thedrow Also, please correct me if I'm understanding wrong - The Controller concept we're introducing in Celery 5 will be platform agnostic, and then we'll provide packages(Platform manager), one of which could be the Kubernetes Operator which will be using that controller internally.

you can currently start with celery 4.4.x and later when the celery 5 enters in rc1 stage we can move the effort to that. (I expect 6-8 months more around)

No actually we're close to releasing an RC. Celery 5 drops Python 2 support which already provides plenty of things that can go wrong and provides a new CLI. We'll get async one Celery 6 and proceed from there.

@brainbreaker If you have experience with Celery and you need that kind of thing at work I suggest you help us. We're working on the next generation of Celery.

I'd love to. I'm not an expert with Celery internals right now but willing to explore and learn. We use it at our work for various asynchronous workloads and there are pain points of managing all the workers, flower, and scaling things across multiple celery use cases.

I am checking your work. feel free to ping if you want me to collaborate on that effort

Thanks, @auvipy. Code is certainly not the best quality right now so please bear with that. I'll definitely need your inputs.

@thedrow Also, please correct me if I'm understanding wrong - The Controller concept we're introducing in Celery 5 will be platform agnostic, and then we'll provide packages(Platform manager), one of which could be the Kubernetes Operator which will be using that controller internally.

I was actually thinking that the Controller becomes a Kubernetes Operator when it is deployed in k8s. You need to install the platform manager for k8s and that's it.

after some discussion with some member here, I realize that to use celery, one has to package its code onto the docker image, so one needs to rebuild for each project/commit the docker image. In this case, maybe all that Celery can do is create a base "abstract" image to be "inherited" by custom-celery-image-with-code that would just do some pip install. Or maybe there is no possible generic "celery" Docker at all...

this is what we should investigate too.

you can easily gitsync your code pretty easily, we do that for airflow.

Is there any official Celery helm chart ? Or Any celery helm chart /Kubernetes yamls that can be used to deploy Celery on Kubernetes ?