Add proxy proposal

[ikaven1024]

Co-authored-by: raymondmiaochaoyue [email protected] Signed-off-by: yingjinhui [email protected]

What type of PR is this? /kind design

What this PR does / why we need it: Support resource aggregation proxy for member clusters.

Which issue(s) this PR fixes: Fixes #

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

NONE

[ikaven1024]

@RainbowMango

[RainbowMango]

Thanks @ikaven1024 !!!

I'll start to look at it ASAP.

[ikaven1024]

Some touches on the expressions.

Thank you for your corrections.

[RainbowMango]

It tells no collisions occur.

This depends on how many replicas are in the Deployment. No collision would happen between different Deployments.

[ikaven1024]

I just look into the generator of pod. Deployment create replicaset with a suffix name genrated by hash of its template, so it will create replicasets with the same name accross clusters. Then replicaset creates pods with random suffix name. The random chars pool size is 27, and suffix length is 5.

According to this article, we can estimate the collision probability by this program:

func main() {
	var k = math.Pow(27, 5)
	fmt.Println("K =", int64(k))
	fmt.Println("replicas\tno_collision_probability")
	for _, n := range []float64{10, 100, 200, 400, 600, 800, 1000, 2000, 4000, 6000, 8000, 10000} {
		fmt.Println(n, "\t", math.Exp(-n*(n-1)/2.0/k))
	}
}

It prints the amount of pods, and the probability of no-collision

K = 14348907
replicas    no_collision_probability
10       0.9999968638775457
100      0.9996550854857799
200      0.998614096037548
400      0.9944540367459195
600      0.9875544919854364
800      0.9779727488347782
1000     0.9657879195731809
2000     0.8699550398230972
4000     0.5726993631642694
6000     0.28529205369204497
8000     0.10754388811853854
10000    0.03067714052002095

It looks well when pod replicas is less than 600. @RainbowMango

[ikaven1024]

The usage of cacher in proxy: https://github.com/ikaven1024/karmada/blob/feature-proxy/pkg/proxy/cache/util.go#L213

[ikaven1024]

Compare between Cacher and Informer:

	Cacher	Informer
how to sync from datasource	List/Watch	List/Watch
Get/List with resourceVersion	support semantics of resourceVersion	equivalent to resourceVersion="0"
List with resourceVersionMatch	suported	not suported
List with labelSelector	suported	suported
List with filedSelector	suported	not suported
watch/ watch with allowWatchBookmarks	suported	not suported

[ikaven1024]

About the question of long pod name

The generated pod name is limited to be no more than 63 chars. @RainbowMango

const (
	// TODO: make this flexible for non-core resources with alternate naming rules.
	maxNameLength          = 63
	randomLength           = 5
	maxGeneratedNameLength = maxNameLength - randomLength
)

func (simpleNameGenerator) GenerateName(base string) string {
	if len(base) > maxGeneratedNameLength {
		base = base[:maxGeneratedNameLength]
	}
	return fmt.Sprintf("%s%s", base, utilrand.String(randomLength))
}

FYI: https://github.com/kubernetes/kubernetes/blob/a3e71b2cefc755bfc83884f00ce11f2b6bdc41c4/staging/src/k8s.io/apiserver/pkg/storage/names/generate.go#L42-L54

Test

1. create a statefulSet with long name in member cluster.

➜  ~ kubectl get deploy
NAME                                                                                                                                                                                                                                            READY   UP-TO-DATE   AVAILABLE   AGE
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa   3/3     3            3           58m
➜  ~ kubectl get po
NAME                                                              READY   STATUS    RESTARTS   AGE
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa88bqq   1/1     Running   0          58m
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8fnhs   1/1     Running   0          58m
aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaajwckc   1/1     Running   0          58m

2. get and edit pod with proxy

➜  kubectl get po aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8fnhs
NAME                                                                    READY   STATUS    RESTARTS   AGE
yjh-2-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8fnhs   1/1     Running   0          59m

➜  kubectl edit po aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8fnhs
pod/yjh-2-aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa8fnhs edited

[RainbowMango]

The generated pod name is limited to be no more than 63 chars.

Great! I just tried on my side, we can create a deployment with name length > 63, but the Pods name indeed <= 63.

deployment.apps/nginx123456789123456789123456789123456789123456789123456789123456789123456789nginx123456789123456789123456789123456789123456789123456789123456789123456789 created
[yangyong@ecs-d8b6 ~]$ kubectl get pods 
NAME                                                              READY   STATUS    RESTARTS   AGE
nginx12345678912345678912345678912345678912345678912345678n9nr6   1/1     Running   0          8s
nginx12345678912345678912345678912345678912345678912345678ptgkz   1/1     Running   0          8s

[ikaven1024]

About separator in name

I prefer to use an invalid char as separator, such as _. It exactly split the cluster name and resource name. @RainbowMango

Test

➜ kubectl get po
NAME                READY   STATUS    RESTARTS   AGE
sts-nginx-0_yjh-1   1/1     Running   0          12m
sts-nginx-1_yjh-1   1/1     Running   0          12m
sts-nginx-2_yjh-1   1/1     Running   0          11m
sts-nginx-0_yjh-2   1/1     Running   0          12m
sts-nginx-1_yjh-2   1/1     Running   0          12m
sts-nginx-2_yjh-2   1/1     Running   0          12m

➜ kubectl get po sts-nginx-0_yjh-1
NAME                READY   STATUS    RESTARTS   AGE
sts-nginx-0_yjh-1   1/1     Running   0          12m

➜ kubectl edit po sts-nginx-0_yjh-1
pod/sts-nginx-0_yjh-1 edited

[RainbowMango]

I prefer to use an invalid char as separator, such as _. It exactly split the cluster name and resource name.

Yeah, that's a great idea, since _ is not a valid DNS character, and it can't be in the name.

[ikaven1024]

@RainbowMango generateName don't help to solve the conflict of names.

I create a deploy and set both metadata.generateName and spec.template.metadata.generateName

apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
  generateName: cluster1-nginx
spec:
  replicas: 3
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      generateName: cluster1-nginx
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx:alpine
        name: nginx

Then create it, and pods are created without cluster prefix

➜ kubectl get po
NAME                     READY   STATUS    RESTARTS   AGE
nginx-547676d98c-btgc7   1/1     Running   0          5s
nginx-547676d98c-nvlsz   1/1     Running   0          5s
nginx-547676d98c-x4ltm   1/1     Running   0          5s

---

Looking into the k8s source, replicaSet's name is generated with deploy.name + hash

	newRS := apps.ReplicaSet{
		ObjectMeta: metav1.ObjectMeta{
			// Make the name deterministic, to ensure idempotence
			Name:            d.Name + "-" + podTemplateSpecHash,
			Namespace:       d.Namespace,
			...
		}
	}

https://github.com/kubernetes/kubernetes/blob/7d8aae51df4f318b71dc3b790dd01d0bcb79c5dd/pkg/controller/deployment/sync.go#L195-L209

Then pod's GenerateName is set to be rs.name-

func GetPodFromTemplate(template *v1.PodTemplateSpec, parentObject runtime.Object, controllerRef *metav1.OwnerReference) (*v1.Pod, error) {
        ...
	prefix := getPodsPrefix(accessor.GetName())

	pod := &v1.Pod{
		ObjectMeta: metav1.ObjectMeta{
			GenerateName: prefix,
                         ...
		},
	}
}

func getPodsPrefix(controllerName string) string {
	// use the dash (if the name isn't too long) to make the pod name a bit prettier
	prefix := fmt.Sprintf("%s-", controllerName)
	if len(validation.ValidatePodName(prefix, true)) != 0 {
		prefix = controllerName
	}
	return prefix
}

https://github.com/kubernetes/kubernetes/blob/78318c7a26928bdabe2491cf022cf9e61be32188/pkg/controller/controller_utils.go#L544-L567

So generateName we set in yaml is not used.

[RainbowMango]

Well, according to the GenerateName comments, seems it will take effect only when Name has not been provided. Just echo the words here:

// GenerateName is an optional prefix, used by the server, to generate a unique // name ONLY IF the Name field has not been provided.

I tried on my side, with the cluster1 as the generatedName:

# cat deployment.yaml 
apiVersion: apps/v1
kind: Deployment
metadata:
  generateName: cluster1
  labels:
    app: nginx
spec:
  replicas: 2
  selector:
    matchLabels:
      app: nginx
  template:
    metadata:
      labels:
        app: nginx
    spec:
      containers:
      - image: nginx
        name: nginx

And I can see the name of Deployment/ReplicaSet/Pod are all starts with generateName(cluster1):

# kubectl get deployments.apps 
NAME            READY   UP-TO-DATE   AVAILABLE   AGE
cluster15t6xh   2/2     2            2           7s

# kubectl get replicasets.apps 
NAME                       DESIRED   CURRENT   READY   AGE
cluster15t6xh-85b98978db   2         2         2       22s

# kubectl get pods 
NAME                             READY   STATUS    RESTARTS   AGE
cluster15t6xh-85b98978db-smdhh   1/1     Running   0          25s
cluster15t6xh-85b98978db-wrdch   1/1     Running   0          25s

[ikaven1024]

Name has not been provided

It seems to be a challenge to karmada. As far as i know, resourceBinding name is created by kind+name. https://github.com/karmada-io/karmada/blob/98ef6281a6731253b68a94551562d1d0c34ab035/pkg/util/names/names.go#L56-L59

[ikaven1024]

@RainbowMango updated

[karmada-bot]

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: RainbowMango

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

• ~~OWNERS~~ [RainbowMango]

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment