Scheduling policies are removed (moved to scheduler config) in k8s v1.23

[ddelange]

/kind bug

1. What kops version are you running? The command kops version, will display this information.

1.22.2

2. What Kubernetes version are you running? kubectl version will print the version if a cluster is running or provide the Kubernetes version specified as a kops flag.

1.22.6

3. What cloud provider are you using?

AWS

4. What commands did you run? What is the simplest way to reproduce this issue?

• I want to run kube-scheduler with the MostAllocated scoring strategy for compatibility with descheduler HighNodeUtilization strategy: https://github.com/kubernetes-sigs/descheduler/tree/v0.23.1#highnodeutilization
• I read the kOps docs on scheduler config (which only mentions usePolicyConfigMap): https://github.com/kubernetes/kops/blob/v1.22.2/docs/cluster_spec.md#kubescheduler
• I read the kubernetes v1.23 docs regarding --policy-configmap and found out it has been deprecated as of v1.23: https://github.com/kubernetes/website/blob/snapshot-initial-v1.23/content/en/docs/reference/scheduling/policies.md
• I checked out the recommended solution using KubeSchedulerConfiguration: https://v1-22.docs.kubernetes.io/docs/reference/scheduling/config/#scheduler-configuration-migrations

5. What happened after the commands executed?

N/A

6. What did you expect to happen?

• I found out that, as far as I can see, kOps v1.23 does not support setting a custom scoring strategy (#12534, #11765) and that new options for the kube-scheduler have been manually added to the codebase until now (e.g. #8407).
• I would like to see more clearly in the docs how to run kOps kube-scheduler with a custom scoring strategy

7. Please provide your cluster manifest. Execute kops get --name my.example.com -o yaml to display your cluster manifest. You may want to remove your cluster name and other sensitive information.

N/A

8. Please run the commands with most verbose logging by adding the -v 10 flag. Paste the logs into this report, or in a gist and provide the gist link here.

N/A

9. Anything else do we need to know?

N/A

[jhohertz]

Also curious about how best to set/influence KubeSchedulerConfiguration, IE setting default topologySpreadConstraints as in: https://kubernetes.io/docs/concepts/workloads/pods/pod-topology-spread-constraints/#cluster-level-default-constraints

[avdhoot]

Currently, we are using the below option/spec to mount policy.cfg into scheduler. As this way is deprecated in k8s 1.23. It will helpful if we have an alternate option in kops 1.23. Without this option, we are stuck with k8s 1.22.

  kubeScheduler:
    usePolicyConfigMap: true

[ddelange]

@rifelpet could you explain what the kind/office-hours tag entails?

[rifelpet]

It just means we'll discuss this issue during the new kOps office hours which is this friday

https://github.com/kubernetes/kops#office-hours

[rifelpet]

We're going to explore having kOps recognize KubeSchedulerConfiguration objects in addition to just Cluster and InstanceGroups

[olemarkus]

We may need to wrap this object as it doesn't support objectmeta. Or we need some sort of other hack to determine what cluster the resource is for.

[justinsb]

I'm proposing that we fix this by supporting scheduling policies as an "additional object", please see https://github.com/kubernetes/kops/pull/13618. The intent is that we get that into 1.25 (but that we don't delay 1.24)

[ddelange]

thanks @justinsb! as an interim solution on our 1.23 cluster: could I simply kubectl apply a KubeSchedulerConfiguration manifest to the running cluster and the scheduler will pick it up without restarts?

[olemarkus]

KubeSchedulerConfiguration is not a resource served by the apiserver. It has to be writted on disk and passed to the scheduler through a CLI flag.

[jonasrydenhag]

Thanks for the fix!

I saw the "Example usage" but how do I apply an KubeSchedulerConfiguration object to a running cluster?

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues and PRs.

This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[ddelange]

Closing as fixed by https://github.com/kubernetes/kops/commit/9661894261630863c191db9619f5e20f41d3f277 as of v1.25.0

[jdomag]

Thanks for the fix!

I saw the "Example usage" but how do I apply an KubeSchedulerConfiguration object to a running cluster?

@jonasrydenhag Have you ever figured this out?

[jonasrydenhag]

Have you ever figured this out?

No. As I remember it, the problem I had at the time was scheduling pods with special hardware requirements on special nodes. So I ended up going with a less dynamic labels and node affinity solution.