Creating a kube-scheduler plugin on Microk8s

[aNDREUET648]

Summary

I'm trying to create a kube-scheduler plugin on Microk8s. To do that I will implement my own scheduler image. I will also need to register, configure the plugin API, implement the extension point interfaces defined in the kubernetes scheduler framework package. As plugins must be compiled with the scheduler, writing a wrapper allows to re-use the scheduler’s code in a clean way. To do all of this I am using an article from medium K8S - Creating a kube-scheduler plugin. The article is truly fantastic and very didactic with a lot of documentation on the subject.

What is my problem?

The problem is that I need to reconfigure the vanilla scheduler and automate the way to deploy it (microk8s scheduler starts as an executable managed by systemd and not as a pod deployment) and the explanations provided by both Julio Renner's article and kubernetes-sigs. Install Scheduler-plugins are very poor and don't fit my needs.

Reproduction Steps

All the sections are very easy to follow until we get to the point near the end of the text Deploying and using the Plugin:

• Build my scheduler container image and upload it to Docker Hub.
• I log into my master node (where the control plane is running).
• I make a backup copy of scheduler.yaml. In my case /var/snap/microk8s/current/args/kube-scheduler.
• Create a file /etc/kubernetes/networktraffic-config.yaml change the values according to your environment. What do I do now?
• Modify the file /etc/kubernetes/manifests/kube-scheduler.yaml to run scheduler-plugins with Network Traffic. And here what do I do

From here I understand that you do something like kubectl apply -f kube-scheduler.yaml -n kube-system

But as we already know this is not the way Configuring MicroK8s services

I hope you can please help me to solve this situation that has been worrying me for a couple of weeks and I can't find the solution. Any help is appreciated.

Best Regards

Andreu

[ktsakalozos]

Hi @aNDREUET648, I followed the instructions in [1] to install the Coscheduling plugin as a second scheduler, here is how:

1. Install MicroK8s and helm3:

sudo snap install microk8s --classic
sudo microk8s status --wait-ready
sudo microk8s enable helm3

2. Install the scheduler:

git clone [email protected]:kubernetes-sigs/scheduler-plugins.git
cd scheduler-plugins/manifests/install/charts
microk8s helm3 install scheduler-plugins as-a-second-scheduler/

3. Verify the scheduler pods are up:

$ microk8s.kubectl get deploy -n scheduler-plugins
NAME                           READY   UP-TO-DATE   AVAILABLE   AGE
scheduler-plugins-scheduler    1/1     1            1           29s
scheduler-plugins-controller   1/1     1            1           29s

4. Follow the instructions in [2] to demo the scheduler.

For running your plugin as a single scheduler we need to clarify how the default scheduler it setup. The scheduler shipped with MicroK8s is a systemd service and is currently not possible (in a trivial way) to start MIcroK8s without starting the scheduler itself.

Also note that the file /var/snap/microk8s/current/args/kube-scheduler is not a yaml file. This file includes the list of arguments we pass to the scheduler, see [3] on what each argument is. To produce a yaml file from the set of arguments you can edit the /var/snap/microk8s/current/args/kube-scheduler file to add a --write-config-to /var/tmp/scheduler.yaml and then do a sudo microk8s stop; sudo microk8s start. The /var/tmp/scheduler.yaml can then be passed to the default argument via the --config argument.

[1] https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md [2] https://github.com/kubernetes-sigs/scheduler-plugins/blob/master/doc/install.md#test-coscheduling [3] https://kubernetes.io/docs/reference/command-line-tools-reference/kube-scheduler/

[aNDREUET648]

Thank you ktsakalozos. I'll try it out and let you know how it worked out for me

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[rubancar]

hey @aNDREUET648 you were able to adjust kube-scheduler using the method that @ktsakalozos mentioned above, I get the configuration yaml and then set the --config parameter pointing to the modified yaml and nothing happens, pods appears not been scheduled by the mode MostAllocated, I'm not quite sure how to get kube-scheduler logs to do a debugging, also microk8s documentation doesn't show anything. Any clue on this