apiserver

apiserver copied to clipboard

Kubernetes Version:

$ kubectl version -o yaml
clientVersion:
  buildDate: "2022-06-15T14:22:29Z"
  compiler: gc
  gitCommit: f66044f4361b9f1f96f0053dd46cb7dce5e990a8
  gitTreeState: clean
  gitVersion: v1.24.2
  goVersion: go1.18.3
  major: "1"
  minor: "24"
  platform: linux/amd64
kustomizeVersion: v4.5.4
serverVersion:
  buildDate: "2022-06-24T09:25:13Z"
  compiler: gc
  gitCommit: efbcd8d250c89b4290356c0fc27843d583c80a35
  gitTreeState: clean
  gitVersion: v1.23.8-gke.400
  goVersion: go1.17.10b7
  major: "1"
  minor: "23"
  platform: linux/amd64

Deployment Setup I had 35 StatefulSets deployed in one namespace as below:

$ kubectl get sts -n yonis-1
NAME                             READY   AGE
elasticsearch-master             1/1     5d21h
sealed-secrets-1-redis-master    1/1     20h
sealed-secrets-10-redis-master   1/1     20h
sealed-secrets-11-redis-master   1/1     20h
sealed-secrets-12-redis-master   1/1     20h
sealed-secrets-13-redis-master   1/1     20h
sealed-secrets-14-redis-master   1/1     20h
sealed-secrets-15-redis-master   1/1     20h
sealed-secrets-16-redis-master   1/1     20h
sealed-secrets-17-redis-master   1/1     20h
sealed-secrets-18-redis-master   1/1     20h
sealed-secrets-19-redis-master   1/1     20h
sealed-secrets-2-redis-master    1/1     20h
sealed-secrets-20-redis-master   1/1     20h
sealed-secrets-21-redis-master   1/1     20h
sealed-secrets-22-redis-master   1/1     20h
sealed-secrets-23-redis-master   1/1     20h
sealed-secrets-24-redis-master   1/1     20h
sealed-secrets-25-redis-master   1/1     20h
sealed-secrets-26-redis-master   1/1     20h
sealed-secrets-27-redis-master   1/1     20h
sealed-secrets-28-redis-master   1/1     20h
sealed-secrets-29-redis-master   1/1     20h
sealed-secrets-3-redis-master    1/1     20h
sealed-secrets-30-redis-master   1/1     20h
sealed-secrets-31-redis-master   1/1     20h
sealed-secrets-32-redis-master   1/1     20h
sealed-secrets-33-redis-master   1/1     20h
sealed-secrets-34-redis-master   1/1     20h
sealed-secrets-4-redis-master    1/1     20h
sealed-secrets-5-redis-master    1/1     20h
sealed-secrets-6-redis-master    1/1     20h
sealed-secrets-7-redis-master    1/1     20h
sealed-secrets-8-redis-master    1/1     20h
sealed-secrets-9-redis-master    1/1     20h

Among all of them, only elasticsearch-master has labels below:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  labels:
    app: elasticsearch-master
    chart: elasticsearch
    data: elasticsearch
...

Other sealed-secrets-{#}-redis-master does not have such labels.

Issue: When I used kubectl with only label selector, everything worked fine:

$ kubectl get sts -n yonis-1 -l app=elasticsearch-master,guardicore.app=infra,data=elasticsearch                                                                            
NAME                   READY   AGE
elasticsearch-master   1/1     5d21h

It worked perfectly as well when I used --raw below:

$ kubectl get --raw "/apis/apps/v1/namespaces/yonis-1/statefulsets?labelSelector=app%3Delasticsearch-master%2Cdata%3Delasticsearch%2Cguardicore.app%3Dinfra"

However, I got empty result when I added additional parameter limit as 15:

$ kubectl get --raw "/apis/apps/v1/namespaces/yonis-1/statefulsets?labelSelector=app%3Delasticsearch-master%2Cdata%3Delasticsearch%2Cguardicore.app%3Dinfra&limit=15"
  "kind": "StatefulSetList",
  "apiVersion": "apps/v1",
  "metadata": {
    "resourceVersion": "3484478"
  },
  "items": []

It does not give the continuous token either. I did more tests, and it turned out that if the condition total resource count > limit * 2 is met, then no result was returned.

For example, I had total 35 StatefulSets, If I gave limit greater than 18, then it returned the correct result. Otherwise, it returned nothing:

• Limit is 18, it returned the StatefulSet correctly: kubectl get --raw "/apis/apps/v1/namespaces/yonis-1/statefulsets?labelSelector=app%3Delasticsearch-master%2Cdata%3Delasticsearch%2Cguardicore.app%3Dinfra&limit=18"
• Limit is 17, nothing was returned: kubectl get --raw "/apis/apps/v1/namespaces/yonis-1/statefulsets?labelSelector=app%3Delasticsearch-master%2Cdata%3Delasticsearch%2Cguardicore.app%3Dinfra&limit=17"

I also used client-go library to tested it, the code look like below:

        cs, _ := clientset.NewForConfig(cfg)

	selector := labels.SelectorFromSet(labels.Set{
		"app":            "elasticsearch-master",
		"guardicore.app": "infra",
		"data":           "elasticsearch",
	})
	stss, _ := cs.AppsV1().StatefulSets("yonis-1").List(ctx, metav1.ListOptions{
		LabelSelector: selector.String(),
		Limit:         15,
	})

The issue persists, so I believe it is likely an issue on apiserver when both 'limit' and 'labelSelector' parameters present.