RFE: 'kubectl evict ...' or similar for conveniently evicting pods

[wking]

trafficstars

What would you like to be added?

A kubectl evict ... subcommand or similar syntactic sugar around the eviction API.

Why is this needed?

Using the delete API is convenient, but dangerous. For example:

$ kubectl -n openshift-monitoring delete pods prometheus-k8s-0 prometheus-k8s-1
pod "prometheus-k8s-0" deleted
pod "prometheus-k8s-1" deleted

Leaves an OpenShift cluster temporarily without monitoring. It's safer to use the eviction API to respect PodDisruptionBudgets, like:

$ kubectl create -f - <<EOF --raw /api/v1/namespaces/openshift-monitoring/pods/prometheus-k8s-0/eviction
> {"apiVersion": "policy/v1", "kind": "Eviction", "metadata": {"name": "prometheus-k8s-0"}}
> EOF
{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","code":201}
$ kubectl create -f - <<EOF --raw /api/v1/namespaces/openshift-monitoring/pods/prometheus-k8s-1/eviction
> {"apiVersion": "policy/v1", "kind": "Eviction", "metadata": {"name": "prometheus-k8s-1"}}
> EOF
Error from server (TooManyRequests): Cannot evict pod as it would violate the pod's disruption budget.

However, that's a bit of a mouthful, and requires a here-doc and duplicating the pod name. It might be possible to add --subresource to the create subcommand to support something like:

$ kubectl -n openshift-monitoring create --subresource eviction pod prometheus-k8s-0

But that would likely bump into the current guards that enforce redundant information between the URI path and the Eviction resource. It's not clear to me why the eviction handler can't backfill missing Eviction resource information like the name from the path information, or really, why the eviction handler cannot default the entire Eviction resource when the caller doesn't need to set explicit delete options. Possibly kubernetes/kubernetes#53185 touches on this, although I haven't wrapped my head around that yet.

Or, instead of trying to make create more flexible, we could grow a new subcommand like evict just for the eviction subresource.

But it would be nice to make the safer eviction API more convenient, so folks didn't have to decide between safety and convenience when bumping pods.

There is at least one existing plugin implementation here.

[k8s-ci-robot]

@wking: This issue is currently awaiting triage.

If a SIG or subproject determines this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wking]

/sig cli

[atiratree]

This was discussed in sig-cli, and the outcome was that this should be adopted by community first as a kubectl krew plugin.

Luckily, it seems there is such a plugin called evict-pod, which has some recognition already: https://github.com/rajatjindal/kubectl-evict-pod.

The repo seems active, but it is missing some features for parametrizing the eviction (most notably gracePeriodSeconds). Hopefully, it should be easy to add by contributing to the repo.

[sftim]

I think this is a kubectl feature request, albeit one we don't plan to prioritize.

/transfer kubectl

[eddiezane]

We would like to see some community demand and interest before considering upstreaming a plugin with this functionality. Please react to the top level issue with a :+1: if you come across this and agree.

[frivoire]

I would be very interested by this feature 😃, here is why:

We run stateful workloads on K8S: typically, a cluster of 3 pods. And our failure tolerance is: loosing 1 pod is ok, but not 2. Sometimes, we need to manually "restart" one precise pod (and not the others), because of a malfunction of that pod.

So our current procedure is :

1. check that all pods are running & ready: either using kubectl get pods -l ... or using monitoring dashboard
2. then execute kubectl delete pod/xxxxxx-i

But of course, this process is dangerous, because of "race condition": => another pod can be deleted (for another reason, cf below) between step 1 & 2 => so, the manual deletion will remove a 2nd pod at the same time => incident 😢

Examples of situation that can create this "race condition":

• a drain of the node (triggered by a K8S cluster scale-down, or for node "upgrade")
• an upgrade of the spec of the pod (a change in the HelmChart is being deployed)
• another engineer doing a similar deletion at the same time

With the proposed feature, our procedure would be simply:

1. execute kubectl evict pod/xxxxxx-i

So, it's now safe (and simpler, but not the objective here) 😃

[sftim]

I'd one day like to be able to evict pods by label. That could arrive first in the plugin, or be something that we implement in-tree.

Other plausible enhancements that build on this basic idea:

• kubectl evict pod --namespace app1 --field-selector spec.nodeName=broken-node.example
• kubectl evict pod --all-namespaces -l foo=bar --max-disruptions=2
• kubectl evict pods/foo pods/bar pods/wibble --grace-period=180

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten