[Question] Deploying resources that need inter-cluster communitation

[bsbds]

Please provide an in-depth description of the question you have:

Hi, I have a question about deploying an resource using karmada that needs inter-cluster communitation.

• What I'm trying to do We have a distributed key-value store cluster, similar to etcd. And we currently use an operator to manage it and want to extend the deployment to multiple clusters using Karmada, distributing cluster nodes across each karmada member cluster.

• The issue The kv store node pods needs to communicate with each other. For deploy via submariner, the operator need to determine dns names of pods across karmada member clusters during reconciliation. More specifically, the operator need the information such as the cluster id and the number of node replicas for each of the karmada member cluster.

However, I have been unable to find a suitable method to pass these informations to member clusters. As these details are defined in PropagationPolicy and used only in Karmada control plane.

What do you think about this question?:

I tried to propagate ResourceBinding CR in Karmada api server to member clusters so that the operator can get the information it needs. But it seems Karmada currently forbids this operation in SkippedResourceConfig.

Does Karmada have some standard support for this kind of configuration? Or are there some alternative solutions? Thanks.

Environment:

• Karmada version: latest
• Kubernetes version: v1.27.3
• Others:

[bsbds]

cc @calvin0327 @RainbowMango

[RainbowMango]

Seems this is the case that split Pods managed by a StatefulSet to multi-clusters. Is that the case?

I tried to propagate ResourceBinding CR in Karmada api server to member clusters so that the operator can get the information it needs. But it seems Karmada currently forbids this operation in SkippedResourceConfig.

Yeah, the ResourceBinding is a kind of internal resource, which is skipped from propagating by default. Let's continue to talk about your use case and see how we can help.

By the way, I might be slow-respond recently due to KubeCon China stuff. If possible, I suggest having a chat at Karmada community meeting(feel free to add an item to the agenda). Sure, discussing it here would be fine as well.

[liangyuanpeng]

Seems this is the case that split Pods managed by a StatefulSet to multi-clusters. Is that the case?

Exactly.

Let me add more information here, This is a question/feature request for project of Xline , Xline is a distributed KV storage engine for multi-cluster, so it will run with a StatefulSet to multi-clusters.

A natural idea is that karmada distributes the scheduling results of multiple clusters to member clusters.

And Other idea is to obtain the corresponding scheduling results from karamda api after scheduling statefulset to multiple clusters, and distribute them to member clusters.

Also want to invite @chaunceyjiang for this issue.

[liangyuanpeng]

May related: https://github.com/karmada-io/karmada/pull/3372

[RainbowMango]

@bsbds as we talked at the community meeting, there are several challenges we need to figure out, such as

• How to split replicas of a StatefulSet into multiple clusters?
• How to let each replica discover and connect to the others?
• maybe more

But I'd say it's an interesting case for Karmada. We can continue the discussion here. Thanks.