openanalytics
New deployment option: Hashicorp Nomad
ShinyProxy already supports many great deployment options, but each is somewhat (subjectively) limited, e.g.
- Single-host versions do not support HA/scaling
- Kubernetes might be overkill for teams that are not using k8s
- ECS is tied to AWS and has limited control over instance types and allocation
- Docker Swarm mode is (IMO) the best fit for lightweight cluster deployments, but it's often referred to as having been taken over by k8s many years ago, and many do not suggest it for new projects
Hashicorp's Nomad is another lightweight, potentially more future-proof option: https://github.com/hashicorp/nomad
Hi @daroczig, nice to hear from you and thank you for the suggestion. It seems Nomad is not open source ("source available"), so it will not be top of our list (unless some organization funds it), but there is good news upcoming with a release later this month that will bring a lot of the comfort of Kubernetes deployments to single-host deployments.
Hi @tverbeke, Thanks for your prompt and kind reply! 🙇
It seems Nomad is not open source ("source available")
Yes, they moved to BSL ~2 years ago, but as per my understanding based on emails with their legal team, it should not be a concern if not building a direct competitor of their service offerings.
so it will not be top of our list (unless some organization funds it)
I totally understand, thank you 👍 Feel free to close the ticket or keep it open, maybe with a "Sponsor needed" label?
but there is good news upcoming with a release later this month that will bring a lot of the comfort of Kubernetes deployments to single-host deployments.
Super exciting, looking forward to learn about the details! Is it something based on minikube, k3s or similar?
Please keep up the great work on ShinyProxy 🙌
Hi @daroczig
Thanks for opening this issue, I'm going to respond a bit in detail, because this is a very interesting discussion and it allows us to explain our views.
I think nomad could fill a gap, where you would need more scaling than a single docker host, but are not ready for k8s. However, from our experience the community has moved to k8s, this also seems to be confirmed by the fact there is (AFAIK) no open-source fork of Nomad, while there are very popular forks of terraform and vault.
Single-host versions do not support HA/scaling
This is definitely true, and I'd love to have some more scaling using Docker. This could potentially be solved inside ShinyProxy, where ShinyProxy is modified to schedule containers on multiple docker nodes. Achieving HA is much more complex, since it would at least require an external HA loadbalancer or using some VIP. Once you have scaling, people probably also want autoscaling. The next thing is they want to have strict network policies between containers. With all these features, it seems we are starting to write our own k8s. Which might work, but also feels counter-productive.
At the same time, we have the experience that using and maintaining k8s is becoming much easier. The last months we did multiple deployments of k3s and it all works very well. Also more and more smaller (and cheaper) cloud providers and bare-metal tools are getting support for k8s.
ECS is tied to AWS
Yes, although we also noticed it's quite popular for ShinyProxy (based on the issues opened here, we don't have telemetry). We are open to add support for similar services at other cloud providers (e.g. Azure and GCP, but definitely also smaller cloud providers).
Docker Swarm mode is (IMO) the best fit for lightweight cluster deployments, but it's often referred to as having been taken over by k8s many years ago, and many do not suggest it for new projects
Yup, and AFAIK Docker as a project is also not adding new features to Swarm. In addition, we have the experience that Swarm has its own issues, which does not necessary make it easy to use.
To conclude, I'd say we have three major deployment options for ShinyProxy:
- single Docker host: very easy to get started, but no scaling/HA
- ECS or similar tools: easy to get started and full scaling/HA
- Kubernetes: more difficult to setup, but almost anything is possible
With Kubernetes getting more feasible to deploy, we also want to look into making using Kubernetes with ShinyProxy easier. That means that as an end-user you have less things to configure. For example, we could look at the operator automatically deploying Redis, the monitoring stack or even network policies.
Super exciting, looking forward to learn about the details! Is it something based on minikube, k3s or similar?
It's not using k8s directly, but we basically added support for docker hosts to the operator, having the same feature w.r.t to updating the config of shinyproxy without downtime, or running multiple shinyporxy instances. In addition, it can automatically deploy Redis, the monitoring stack, setup TLS, templates etc.
The idea is that with a single (small) docker-compose.yml file you get a full production-ready ShinyProxy server.
This is also ready being used at multiple customers, and we plan to release it in the coming weeks.
Finally, I just wanted to mention that our work on ShinyProxy is really focused on what the community needs. Therefore, we really like when an issue like this is opened and we invite anyone to share their experiences and any feature request.
I think nomad could fill a gap, where you would need more scaling than a single docker host, but are not ready for k8s. However, from our experience the community has moved to k8s, this also seems to be confirmed by the fact there is (AFAIK) no open-source fork of Nomad, while there are very popular forks of terraform and vault.
Agreed, unfortunately 😓
I'd love to have some more scaling using Docker.
Docker Swarm mode was intended for that purpose, right? :)
It can help not only with scaling, but HA as well, and overlay networks (over a private network between nodes e.g. at AWS or Hetzner) can somewhat solve for "strict network policies".
But you mentioned "we have the experience that Swarm has its own issues" -- can you please share more details? I'm super interested, as I'm currently considering going with Swarm for a pilot project instead of k3s for simplicity.
With Kubernetes getting more feasible to deploy, we also want to look into making using Kubernetes with ShinyProxy easier. That means that as an end-user you have less things to configure. For example, we could look at the operator automatically deploying Redis, the monitoring stack or even network policies.
That sounds very promising, thank you!! I'm really looking forward to the new Kubernetes Operator release 🙇
Out of curiosity, are you using Caddy for the reverse proxy with TLS?
Finally, I just wanted to mention that our work on ShinyProxy is really focused on what the community needs. Therefore, we really like when an issue like this is opened and we invite anyone to share their experiences and any feature request.
What I'm currently experimenting with is how to find the best cloud instance types to be dynamically added to the Shinyproxy pool -- preferably via Swarm, Nomad, or k3s. So this involves monitoring, starting nodes and joining the cluster then getting rid of unused servers etc. For context, this is as part of my @spareCores efforts 😊
It can help not only with scaling, but HA as well, and overlay networks (over a private network between nodes e.g. at AWS or Hetzner) can somewhat solve for "strict network policies".
With strict network policies I meant that you cannot easily isolate traffic between containers. So that the container of app X cannot reach the container of app Y.
But you mentioned "we have the experience that Swarm has its own issues" -- can you please share more details? I'm super interested, as I'm currently considering going with Swarm for a pilot project instead of k3s for simplicity.
The biggest issue is https://github.com/openanalytics/shinyproxy/issues/399 : basically how docker swarm pulls images is different from regular docker and we didn't find an option to improve it. In general it just seems swarm is less polished then regular docker and we don't expect docker to still improve swarm.
Out of curiosity, are you using Caddy for the reverse proxy with TLS?
Yes, good guess! Caddy is very good at this, so we didn't want to re-invent the wheel.
Thank you so much for the further context on Docker Swarm, super useful! 🙇