What Would You Like to Be Added

In the code of karmada-scheduler establish connection to karmada-scheduler-estimator, add the logic of handling ExternalName type Service. This will allow the connection establishment to work without kube-proxy.

Why is This Needed

There should be two different ways to deploy Karmada:

1. Deploy in Physical Machine or Virtual Machine directly.
2. Deploy in Host Kubernetes Cluster.

Currently, the connection establishment rely on kube-proxy to do the network traffic reforwarding. But in karmada-scheduler deployed directly in physical machine without host kubernetes cluster, there is no kube-proxy.

Current Logic

karmada-scheduler needs host and port information to connect to karmada-scheduler-estimator.

Host: It's generated from cluster name. For example, if the cluster name is "ray-karmada-member1", the host will be "karmada-scheduler-estimator-ray-karmada-member1".

Port: "--scheduler-estimator-port" parameter of "karmada-scheduler"

Relevant code: https://github.com/karmada-io/karmada/blob/665af6916a733da4cae2ce1dfb9321c264f114ff/pkg/estimator/client/cache.go#L82

hack/deploy-scheduler-estimator.sh will deploy a Service named "karmada-scheduler-estimator-{{member_cluster_name}}". So if Karmada is deployed inside host kubernetes cluster, the kube-proxy will implement a form of virtual IP for the Service.

apiVersion: v1
kind: Service
metadata:
  name: karmada-scheduler-estimator-{{member_cluster_name}}
  namespace: karmada-system
  labels:
    cluster: {{member_cluster_name}}
spec:
  selector:
    app: karmada-scheduler-estimator-{{member_cluster_name}}
  ports:
    - protocol: TCP
      port: 10352
      targetPort: 10352
  type: ClusterIP

How to Do It

In physical machine deployment, we don't have kube-proxy. We use another kind of Virual IP to forward traffic, for example Linux Virtual Server.

But LVS will not install iptables rules in the machine deploying karmada-scheduler. So we need karmada-scheduler itself to interpret "Service" content to connect to the required Virtual IP and Port provided by LVS server.

There are existing logic used by Kubernetes API Server to connect to Web Hook: https://github.com/kubernetes/kubernetes/blob/8287e21228e9b1c945d031581e2fc63920f46d56/staging/src/k8s.io/apiserver/pkg/util/proxy/proxy.go#L92

We will use similar but different logic to support Physical Machine Deployment.

Design Details

The Service in Physical Machine Deployment will be ExternalName type like the following.

apiVersion: v1
kind: Service
metadata:
  name: karmada-scheduler-estimator-{{member_cluster_name}}
  namespace: karmada-system
  labels:
    cluster: {{member_cluster_name}}
spec:
  ports:
    - protocol: TCP
      port: 10352
      targetPort: 12345
  type: ExternalName
  externalName: 10.0.1.2

spec.ports[0].port: This is the same as "--scheduler-estimator-port" parameter of "karmada-scheduler".

spec.ports[0].targetPort: This is the Virtual Port provided by LVS.

spec.externalName: This is the Virtual IP provided by LVS.

Note: If this is a test cluster deployed on your Development Machine, you can also assign real IP and port to avoid LVS.

/assign

I missed this from my radar. Would like to talk about it at the next community meeting? (if so, please remember to add an agenda)

I missed this from my radar. Would like to talk about it at the next community meeting? (if so, please remember to add an agenda)

Do you want me to tell why we need to deploy Karmada in Physical Machine directly without relying on host Kubernetes Cluster?

Sounds like a reasonable feature, but I'm not sure the reason why deploy Karmada in Physical Machine. What's the use case here? also, cc estimator author @Garrybest for comments.

I believe you should use headless service if you have a external IP for estimator.

Here is what I found.

Note: ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider using headless Services.

AFIAK, when you specify externalName as a IP address, the resolve will not take effect as it is not a CNAME record. I'm afraid it's correct to use a headless service and create a endpoint manually here. Here is some information that may be helpful.

@Garrybest

I believe you should use headless service if you have a external IP for estimator.

Here is what I found.

Note: ExternalName accepts an IPv4 address string, but as a DNS name comprised of digits, not as an IP address. ExternalNames that resemble IPv4 addresses are not resolved by CoreDNS or ingress-nginx because ExternalName is intended to specify a canonical DNS name. To hardcode an IP address, consider using headless Services.

AFIAK, when you specify externalName as a IP address, the resolve will not take effect as it is not a CNAME record. I'm afraid it's correct to use a headless service and create a endpoint manually here. Here is some information that may be helpful.

(1) We deploy Karmada in Physical Machine directly. => No Host Kubernetes Cluster. => No CoreDNS. => DNS is not an issue since it does not exist.

(2) If I use headless Service, there will need to be an Endpoints object. Endpoints can specify multiple IP Addresses, so how should I choose one IP? Implementing Load Balancing in karmada-scheduler & karmada-descheduler seems too cumbersome. Currently we use LVS to do Load Balancing, there will be only one IP Address. Use ExternalName will be much easier.

(3) There are existing logic used by Kubernetes API Server to connect to Web Hook: https://github.com/kubernetes/kubernetes/blob/8287e21228e9b1c945d031581e2fc63920f46d56/staging/src/k8s.io/apiserver/pkg/util/proxy/proxy.go#L92

The related issue: aggregator: unify resolver implementation and tests #46623

I think use similar logic will suffice.