sharding with a deployment with '--resources=pods' and '--node=""' does not fetch pending pods

[carlosrmendes]

What happened: a deployment with --resources=pods and --node="" does not fetch pending pods

What you expected to happen: pending pods be fetched by ksm

How to reproduce it (as minimally and precisely as possible):

• create a ksm deploy with --resources=pods and --node="" and -v=10
• force another pod to be in a pending state
• check the logs for the api request like https://<kubernetes_api_ip>:443/api/v1/pods?allowWatchBookmarks=true&fieldSelector=spec.nodeName%3D%22%22&resourceVersion=3998681&timeout=5m23s&timeoutSeconds=323&watch=true
• 'kubect proxy' and try to make the same request
• make the same request without the &fieldSelector=spec.nodeName%3D%22%22 and confirm that on the response, the pending pods does not have any nodeSelectot key with an empty value

• kube-state-metrics version: v2.10.1
• Kubernetes version (use kubectl version): k3s v1.24.10

[k8s-ci-robot]

This issue is currently awaiting triage.

If kube-state-metrics contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[CatherineF-dev]

Thanks for reporting this.

Could you help paste your pending pod yaml, if it's available? Want to see nodeName value on the yaml

[CatherineF-dev]

/assign @CatherineF-dev

[carlosrmendes]

The pending pod doesn't even have the nodeName key, probably the key is only set after the pod is scheduled to a node.

apiVersion: v1
kind: Pod
metadata:
  creationTimestamp: "2024-03-27T19:56:40Z"
  generateName: pending-test-
  labels:
    app.kubernetes.io/name: pending-test
    controller-revision-hash: pending-test-675ffcc546
    statefulset.kubernetes.io/pod-name: pending-test-0
  name: pending-test-0
  namespace: default
  ownerReferences:
  - apiVersion: apps/v1
    blockOwnerDeletion: true
    controller: true
    kind: StatefulSet
    name: pending-test
    uid: c5ed648f-afaa-44ef-9652-40b4633ca5b9
  resourceVersion: "3998134"
  uid: a04b2548-6977-4749-8375-96489b1aa42a
spec:
  containers:
  - image: nginx
    imagePullPolicy: IfNotPresent
    name: pending-test
    resources:
      requests:
        memory: 64Gi
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-ndgdz
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  hostname: pending-test-0
  preemptionPolicy: PreemptLowerPriority
  priority: 2000001000
  priorityClassName: system-node-critical
  restartPolicy: Always
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  subdomain: pending-test
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - name: kube-api-access-ndgdz
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2024-03-27T19:56:40Z"
    message: '0/1 nodes are available: 1 Insufficient memory. preemption: 0/1 nodes
      are available: 1 Insufficient memory.'
    reason: Unschedulable
    status: "False"
    type: PodScheduled
  phase: Pending
  qosClass: Burstable

[CatherineF-dev]

Wondering whether apiserver fieldSelector supports -nodeName which means nodeName doesn't exist

[CatherineF-dev]

I tried again and verified --field-selector spec.nodeName= can get unscheduled pods.

--node="" is translated into spec.nodeName=

1. nothing: get all pods
2. --node="A": --field-selector spec.nodeName=A get pods on node A
3. --node="": --field-selector spec.nodeName= get pods with no node assigned

kubectl get pods --field-selector spec.nodeName= -A

NAMESPACE   NAME                          READY   STATUS    RESTARTS   AGE
default     helloworld-7f5cd44b59-zhgzj   0/1     Pending   0          87s

kube_pod_status_scheduled{namespace="default",pod="helloworld-7f5cd44b59-zhgzj",uid="26018553-ceb6-427f-be1a-088032edac7a",condition="false"} 1

[carlosrmendes]

with kubectl works, the Pending pods are returned, but making an http request to the kubernetes api (same way that ksm do) does not:

[carlosrmendes]

I figured it out that the problem is the --node="" translation to url parameter fieldSelector=spec.nodeName%3D%22%22, without the encoded quotes works: fieldSelector=spec.nodeName%3D

[CatherineF-dev]

@carlosrmendes does it mean we need to fix this for KSM? Or just your local query doesn't work.

[CatherineF-dev]

I can reproduce it that --node="" doesn't work as expected.

/usr/lib/go-1.22/bin/go  run main.go  --resources=pods --node=  --kubeconfig ~/.kube/config

curl localhost:8080/metrics | grep kube_pod_status