Docker containerization

[SiqingYu]

We have a WIP pull request to containerize mygpo for a local development environment - https://github.com/gpodder/mygpo/pull/32

Containerization is a relatively massive undertaking that requires weeks, if not months of effort. We'll take a progressive approach to bring containers onto our production environment.

We will migrate our production environment to a containerized solution before addressing at least the following issues:

• [ ] Local development environment
• [ ] Continuous integration
• [ ] Application migration from vanilla Linux servers to containers
• [ ] Deployment strategy (e.g., rolling updates)

For the moment our server setup on prgmr.com is below:

• App server for our Nginx, Gunicorn, Django app - 4 vCPUs, 4 GB memory.
• Database server for our PostgreSQL - 4 vCPUs, 2 GB memory.

The database server is not our primary target for now because we rarely upgrade the database while we regularly deploy the latest master codebase to the app server. Database containerization is much more complicated than stateless web applications. Fortunately, I don't see any direct benefit of a database container for us.

A single app server might not be ideal. As you may know, Kubernetes requires at least three servers for high availability. Meanwhile, lightweight Kubernetes variants focusing on single-node deployment like k8s might be helpful. We might also ask for more resources from prgmr.com or other cloud providers.

For more info about lightweight Kubernetes: K3s, minikube or microk8s? : kubernetes

Why container orchestration?

There are many benefits to enumerate, all of which boil down to smooth application deployment. Our existing infrastructure has many moving parts, so the deployment is error-prone. I have deployed an update last week and caused a half-an-hour outage because of Gunicorn misconfiguration. Not cool; my fault :pensive:.

Why Kubernetes?

• I'm not opposed to Docker Swarm and others. I'm inclined to use Kubernetes because I have used Kubernetes but don't know much about Swarm.
• Although Kubernetes is well-known for its steep learning curve while Swarm is relatively more accessible for new-comers, the main difficulty is on the cluster setup. For development uses the cluster choice is less critical. Kubernetes also has easy-to-use local development variants like Minikube.
• Kubernetes has good documentation and comes with many out-of-box solutions.
• There are many managed Kubernetes cloud providers, which is great if we want to relieve the burden of administration and maintenance in the future.

[SiqingYu]

Your advice would be appreciated. :smile: @stefankoegl @JeanFred @morgenroth @fabolhak @elelay

[stefankoegl]

I think I have a solid understanding of what containerization is, but I have never worked with kubernetes myself. We should keep in mind that we don't make it too complicated to get a test system up and running, to ensure that new contributors can join easily.

More generally, I think gpodder.net's architecture allows for it to be split up into micro services if that's what you're going for. I'm happy to discuss that further (eg on Slack).

[ryoung29]

I can help with this as well

[SiqingYu]

@ryoung29 Thanks. There is a WIP PR that you can follow. https://github.com/gpodder/mygpo/pull/32

[zb3]

I'm currently trying to set up a development environment using docker compose (I'm learning and my goal is to do docker-compose up and have it running).

I saw issue #262 and looked into PR #32, and it appears to me that containerization here is ultimately concerned with the production use (as opposed to #262). The development setup I'm currently trying to create has:

• DEBUG=true and using manage.py runserver (exceptions are visible and static files are served)
• source directory mounted as a volume so no need to rebuild the image to test changes
• test dependencies installed
• remote debugging support

Since my points above seem to conflict with production use, maybe there could be a separate docker image and compose file for the development setup? If so, I might be able to help with that once I get that running properly.

[oliverkane]

Hi there. I found this Issue while I was on my way to dockerize this myself since I didn't see anything related to that in the master branch. Glad to see this is in progress. What's the best way for me to connect to folks and see where I could best contribute to this effort? My end goal is to get this into k8s, and I'm pretty familiar with dockerizing apps, though I'm a bit poor at python.

[PascalNoisette]

Hi,

I came accross the april 2021 fork of @maxolasersquad It is running quite well out of the box and I think it solve #32 and partially this issue (not the k8s part, it's implemented with docker-compose).

@zb3, I am running behind a nginx reverse proxy with DEBUG: "True" the image: maxolasersquad/mygpo so you might be interested as well. (celery must be restarted after postgres though and after migrate, otherwise I get some Relation "django_celery_beat_periodictask" does not exist).

Regards,

[nagyv]

I created a dockerized version that I use in Kubernetes. It's available at https://gitlab.com/nagyv/gpodder

Please, pick any of my code and contribute it here.

[almereyda]

@nagyv FYI the repository https://gitlab.com/nagyv/gpodder is 404 nowadays. I suspect it is superseded by

• #809

[nagyv]

@almereyda Correct. the PR includes minimal docs too and it would add all the changes I made to the main mygpo repo.

[albertopasqualetto]

Is there any update or ETA?

[Ryochan7]

Closest thing you will get would be at https://codeberg.org/atomdmac/mygpo-docker/. It allows running the mygpo project and related dependencies in Docker.