Update k8s resource configurations

[t3hmrman]

After recently getting zulip up on my own cluster, the resource configurations I hacked a little bit to get working are in this PR.

There probably needs to be a lot of discussion on how to integrate these with docker-zulip properly -- for example helm I think is a step too far as a requirement -- but this should be a starting point to see how I broke apart the single file.

[t3hmrman]

Please feel free to let me know if I should make an actual issue for this -- there isn't one right now but I made this PR based off of the discussion in the reverse proxy issue

I'd love some feedback on what to change to make this more palatable for docker-zulip

[timabbott]

I'm not an expert on kubernetes; @galexrt do you have an opinion on this direction?

Or alternatively, @t3hmrman, if you point to some resources and/or explain your reasoning for these changes, I can review this after doing a bit of reading.

[timabbott]

And I should add that we could definitely use help with polishing the Docker/k8s setup in this project if you have the time and interest to do so.

[t3hmrman]

I'd love to help! It took me an inordinate amount of time to get to putting this PR up, but I'm expecting there to be a bunch of discussion in here on how to actually do this properly.

Here's a cursory explanation of the reasoning:

• ReplicationControllers are still around but Deployments (that manage ReplicationControllers underneath) are the preferred way of doing things these days (you can see this in the documentation for ReplicationController`)
• I thought adding versioning (based on k8s) is a good way to keep in step with the different changes to the k8s API without breaking people using older code. For example, if anyone is using a version of k8s before Deployments were GA, then I wouldn't want to break them
• Splitting up the files I think makes it easier to tell what a zulip "deployment" consists of, and it somewhat focuses the tweaking (if necessary), if you know that you
• Addition of PersistentVolumeClaims (*.pvc.yaml) to manage data -- it's generally better to use them and leave management of space to another entity (whether it's a tool like Rook or OpenEBS or some cloud provider)
• I avoided a helm chart because I think helm is more complexity than it's worth and is actually bad for early kubernetes practitioners -- I hear things are getting better w/ Helm 3 but for relatively simple deployments (of which I consider Zulip one), I find it's much better for understanding to see all the pieces and files as simply as possible to enhance at least the feeling of understanding what's going on. All this said, I'm totally fine with adding <k8s version>/helm and <k8s version>/manual repositories, or expecting people to adopt a tool like kustomize which is less complex than helm. I personally "pioneered" the make infra pattern so that's how I'd write it if given the opportunity, but it's not a widely enough known approach for a big project like zulip (helm/kustomize/other options would be better).

The choices I've made are certainly non-standard, but I think they're valuable because they're a bit easier to approach than one gigantic file (which is what people do -- a lot of the time people don't expect end users to look at the constituent parts of the software). For users that just want to kubectl apply -f <one file> or even curl ... | kubectl apply then I'm also fine with creating a file that is everything combined as well.

[timabbott]

Awesome, this is much appreciated! I likely won't be able to dig into this stuff this week due to the New Years' holiday, but I'm hoping to do so soon.

[t3hmrman]

Somewhat related, but since kustomize was been merged into kubectl I think it would be a fantastic to use in this folder, with a simple Makefile in the folder (as an alternative to helm).

One of the best parts of helm was the centralization of the parts of the configuration that changed, and customize solves that without imposing the rest of helm. One of the worst parts of my non-standard approach was having to do all the wiring of variables into the files with tools like envsubst but now kubectl kustomize will make things easier.

[philband]

How is the state on this PR? I am also currently figuring out how to get Zulip properly running in Kubernetes. @t3hmrman would you be open to some suggestions from my setup to improve the PR?

[t3hmrman]

Hey @philband sure -- this PR hasn't been touched in a while, but it's how I'm running zulip and haven't had any problems with it recently.

What I need to add to this is use of kustomize, which I think I've adapted my own setup to use

[philband]

Hi @t3hmrman, good to hear. I have committed base my current work on top of yours, so you can easily see a diff: https://github.com/philband/docker-zulip/tree/update-k8s-config https://github.com/philband/docker-zulip/commit/d07d661fd3071fcae8e8404c9258bf44e54b2fc7

Notable changes:

• The Deployments for PostgreSQL and RabbitMQ were missing, added my versions for those
• Migrated all the secrets to a Secrets definition, referencing it from the Deployments
• Changed images/initialization for memcached & redis to be in line with the docker-compose setup
  • Using password authentication, as in the current zulip upstream version
  • Image updated to the official, lightweight alpine images
• Remove unnecessary/unused memached PersistentVolumeClaim
• Refactored/unified labels & selectors

Please let me know, what you think about these changes.

[t3hmrman]

Hey @philband the changes look good to me -- the only thing I'm thinking of adding to that is kustomize -- any thoughts? I can go ahead and pull the changes you made into this PR as they are though

[puckettgw]

I'm kind of disappointed with the decision to not make a helm3 chart.

I don't quite agree with the reasoning ... if anything, "simple" deployments (I disagree that something with 5 dependencies is simple, but let's sidestep that argument) would be a great way for helm newbies to get their feet wet.

Also, I'm not sure how installing the helm3 binary, editing a values.yaml file and typing helm install zulip . is too difficult. You really don't have to know anything about how helm works to install a well-factored chart; you just need to know how to edit yaml files.

[t3hmrman]

Hey @puckettgw that is absolutely not a decision taken by the docker-zulip maintainers! I'm not a docker-zulip maintainer and you are absolutely free to make a Helm 3 (or helm 2, etc) chart!

I don't want to pass off my opinions on the matter as fact or even the probable direction of this project -- these updates were meant to help refresh the configs that were already there, no harm in adding a helm directory and adding helm deployments there

[puckettgw]

Hey @t3hmrman That's totally reasonable. I see now that I was a bit hasty in my gripe. Thanks for the reply!

[t3hmrman]

Oh @puckettgw no worries! This PR itself is actually a bit stale now in and of itself -- I need to get around to merging in the changes that were suggested, and get it on track to use kustomize as well...

[ayushin]

Is there any work on this?

We are trying to run zulip in production and the current setup with supervisor in docker is really really outdated (we run k8s on multiple providers with terraform).

I am not sure what the right way forward should be as the super-hairy entrypoint parsing env variables into files could be replaced with kubernettes configmaps and secrets, but also all the processes would need to be split into separate docker images.

@timabbott are there any plans for creating a more modern deployment setup perhaps with k8s helm and a terraform module out of the box?

I understand that changes things completely for the entire dev team, but if you don't do this at some point this will make the whole project dev-ops hopelessly outdated and increasingly difficult to maintain.

[t3hmrman]

I haven't touched this or thought about it in a really long time but I can also say that Helm 3 is almost certainly the way to go now for easy installs of things, especially since the tiller requirement is now gone. Either a directory that is meant to be run with kustomize or helm 3 (and probably helm is the better choice there).

[klardotsh]

Hey folks, looks like this has been sitting open for over two years without comment: at this point would it make more sense to try to adopt the work and continue it, or to close as stale/outdated and of course allow new work in this space to spin up as desired? I come from a long background of containerized deployments but the Kubernetes world is not my mother tongue so I'm not entirely up to date on what Helm versions or patterns are common.

[t3hmrman]

Hey @klardotsh I think it's probably a good idea to abandon this and go with Helm 3 -- I still prefer kustomize (it's built in now) but Helm 3 is the most common choice and is what people are likely to be familiar with, with good answers for things like variable substitution.

I'm going to close this PR since it's gotten so old at this point, and probably wrong in large parts.

[ayushin]

We use kubernetes for django deployment (also possible with side containers) directly by using terraform.

Maybe this works for zulip too:

https://github.com/djangoflow/terraform-kubernetes-django

(no need for helm or kustomize)