skyportal
skyportal copied to clipboard
skyportal
DRAFT/WIP: Kubernetes integration
Hi @stefanv, it's a little bit messy, but these are the main changes I made when trying to get SP to run in kubernetes.
This did get me to successfully run the app, with the exception that this model has one pod per service (and instance of the app), which is way to much.
As we discussed, instead we probably want a pod with all the essential microservices (that don't need to scale) and maube even some instances of the app, and just be able to spawn more instances of the app only in other containers.
What you are looking for is mostly what is added in the kubernetes directory, as well as the kubernetes_setup.py file.
What is also missing is the auto scaling. It's very easy to tell the k8 config to autoscale, but I haven't implemented what is needed on the nginx side to refresh its config and point to new (up and running) instances. I haven't found a way to do that in the config itself, but maybe a seperate cronjob that interrogates the health monitor to find the list of healthy instances and their IP addresses (or k8 service host name), updates the config and restart nginx (as graciously as possible).
Again, this is very VERY much a draft, as I learned a lot of kubernetes pretty much while writing this. However, I think that what the kubernetes_setup.py script does is what we want, and tweaking it could yield the results we want.
@Theodlz Thanks for sharing this. I was hoping to make the minimal adjustments necessary to the fritz deploy repo. Is there a reason we'd want to start from scratch here?
@Theodlz Thanks for sharing this. I was hoping to make the minimal adjustments necessary to the fritz deploy repo. Is there a reason we'd want to start from scratch here?
It could be in the fritz-deploy repo too, but I'm sure you'll need changes to be made here as well if you want pods to talk to each other (one good example being the websockets server). Right now a lot of things in SP and baselayer listen on localhost, or on ports that won't be accessible by other pods.
But of course, aside from changes required here, we'll have some to make in the deploy script. For me, baselayer and SkyPortal as is (without some of these changes) couldn't really run with kubernetes in the context of having multiple pods for different services/apps.
Ideally, we could avoid duplication, so either have the fritz-deploy repo content migrate here, or the other way around, with one copy of the configuration files, customized for different types of deploys.
@stefanv I think it's fine to have all or most of the kubernetes stuff in fritz-deploy.
But here, we still need to have the changes required in the multiple microservices and config files to allow us to specify different hosts than localhost where needed, which shouldn't be too many changes.
To keep all information together, let's refactor the configuration so that we have a host and port for each microservice and service.
services:
slack:
port: 64100
external_services:
lt:
host: 161....
port: 80
To keep all information together, let's refactor the configuration so that we have a host and port for each microservice and service.
services: slack: port: 64100 external_services: lt: host: 161.... port: 80
Yes that sounds good! Easier to read than having 2 blocks for hosts and ports like I do right now.
@stefanv do you want me to take care of that SP-side PR? Happy to re-open a "cleaner" one (leave this one as a draft for reference) to take care of these hosts lines and other things we might need for kubernetes to do its job properly.
As you wish, Theo. I have some cycles to work on it too, just let me know. I think smaller, focused PRs are the way to go as long as we can get away with it! :+1:
As you wish, Theo. I have some cycles to work on it too, just let me know. I think smaller, focused PRs are the way to go as long as we can get away with it! 👍
Ok! I am busy with some PRs on Kowalski at the moment, so I guess it depends on how busy you are as well. I can probably open that first PR tomorrow, but not today. If you want to get started now, I won't stop you!