karmada icon indicating copy to clipboard operation
karmada copied to clipboard

Published 20 hours ago • verified publisher icon karmada-io

Improve cache with structed objects

Open ikaven1024 opened this issue 1 year ago • 25 comments

What type of PR is this? /kind flake

What this PR does / why we need it:

Which issue(s) this PR fixes: Fixes https://github.com/karmada-io/karmada/issues/4405#issuecomment-1853367471

Special notes for your reviewer:

Does this PR introduce a user-facing change?:

`karmada-search`: improve with structed objects in cache.

ikaven1024 avatar Dec 15 '23 16:12 ikaven1024

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: To complete the pull request process, please ask for approval from ikaven1024 after the PR has been reviewed.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment Approvers can cancel approval by writing /approve cancel in a comment

karmada-bot avatar Dec 15 '23 16:12 karmada-bot

Hi @xigang , I have changed to structed objects in cache, and you can request in pb type. Set it in your client's restconfig:

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

And you will check it with --v=8. e.g.:

I1216 00:10:29.331099    5097 round_trippers.go:469] Request Headers:
I1216 00:10:29.331104    5097 round_trippers.go:473]     Accept: application/vnd.kubernetes.protobuf,application/json
I1216 00:10:29.331108    5097 round_trippers.go:473]     User-Agent: ___1go_build_github_com_karmada_io_karmada__output/v0.0.0 (darwin/arm64) kubernetes/$Format
I1216 00:10:29.570068    5097 round_trippers.go:574] Response Status: 200 OK in 238 milliseconds
I1216 00:10:29.570096    5097 round_trippers.go:577] Response Headers:
I1216 00:10:29.570101    5097 round_trippers.go:580]     Audit-Id: 461a6a5d-09ae-4449-9021-9fc1d8cc3cb1
I1216 00:10:29.570111    5097 round_trippers.go:580]     Audit-Id: 461a6a5d-09ae-4449-9021-9fc1d8cc3cb1
I1216 00:10:29.570119    5097 round_trippers.go:580]     Cache-Control: no-cache, private
I1216 00:10:29.570122    5097 round_trippers.go:580]     Cache-Control: no-cache, private
I1216 00:10:29.570126    5097 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf

Would you test the performance with it again?

ikaven1024 avatar Dec 15 '23 16:12 ikaven1024

Hi @xigang , I have changed to structed objects in cache, and you can request in pb type. Set it in your client's restconfig:

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

And you will check it with --v=8. e.g.:

I1216 00:10:29.331099    5097 round_trippers.go:469] Request Headers:
I1216 00:10:29.331104    5097 round_trippers.go:473]     Accept: application/vnd.kubernetes.protobuf,application/json
I1216 00:10:29.331108    5097 round_trippers.go:473]     User-Agent: ___1go_build_github_com_karmada_io_karmada__output/v0.0.0 (darwin/arm64) kubernetes/$Format
I1216 00:10:29.570068    5097 round_trippers.go:574] Response Status: 200 OK in 238 milliseconds
I1216 00:10:29.570096    5097 round_trippers.go:577] Response Headers:
I1216 00:10:29.570101    5097 round_trippers.go:580]     Audit-Id: 461a6a5d-09ae-4449-9021-9fc1d8cc3cb1
I1216 00:10:29.570111    5097 round_trippers.go:580]     Audit-Id: 461a6a5d-09ae-4449-9021-9fc1d8cc3cb1
I1216 00:10:29.570119    5097 round_trippers.go:580]     Cache-Control: no-cache, private
I1216 00:10:29.570122    5097 round_trippers.go:580]     Cache-Control: no-cache, private
I1216 00:10:29.570126    5097 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf

Would you test the performance with it again?

@ikaven1024 good, I will test the performance again later.

xigang avatar Dec 16 '23 08:12 xigang

@ikaven1024 In a federated cluster with 5.6W nodes, List 91W pods request test results:

Pod Total Transfer Protocol RT
91W application/json 11m3s
91W application/vnd.kubernetes.protobuf 3m34s

if the federated cluster scale target is 10W nodes and 200W pods, the pb type will take about 7min, which is still unfriendly to the client.

if the server side does data sharding by index and the client concurrently requests List. the client will load all the data in about 1min.

xigang avatar Dec 20 '23 14:12 xigang

@ikaven1024 If protocol buffer are used, CRD is not supported. there will be this flaw.

xigang avatar Jan 04 '24 09:01 xigang 

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

If you set AcceptContentTypes as application/vnd.kubernetes.protobuf,application/json, response type should be adaptive, pb for intree objects, and json for crs.

Have you actually encountered this problem？

ikaven1024 avatar Jan 04 '24 10:01 ikaven1024 

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

If you set AcceptContentTypes as application/vnd.kubernetes.protobuf,application/json, response type should be adaptive, pb for intree objects, and json for crs.

Have you actually encountered this problem？

@ikaven1024 Yes, I encountered this problem while testing CRD resources.

failed to list *v1alpha1.DuplicateSet: object *unstructured.UnstructuredList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message

xigang avatar Jan 05 '24 02:01 xigang 

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

If you set AcceptContentTypes as application/vnd.kubernetes.protobuf,application/json, response type should be adaptive, pb for intree objects, and json for crs. Have you actually encountered this problem？

@ikaven1024 Yes, I encountered this problem while testing CRD resources.

failed to list *v1alpha1.DuplicateSet: object *unstructured.UnstructuredList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message

Would you log the request headers with --v=8?

ikaven1024 avatar Jan 05 '24 02:01 ikaven1024

@ikaven1024 I used CRD typed client for testing. the client requests headers:

I0105 11:53:39.844297 1315213 round_trippers.go:182] Request Headers:
I0105 11:53:39.844308 1315213 round_trippers.go:186]     Accept: application/vnd.kubernetes.protobuf,application/json
I0105 11:53:39.844316 1315213 round_trippers.go:186]     User-Agent: podinformer/v0.0.0 (linux/amd64) kubernetes/$Format
W0105 11:53:39.853644 1315213 reflector.go:424] github.com/cloud/beirut-api/pkg/client/informers/externalversions/factory.go:116: failed to list *v1alpha1.DuplicateSet: object *unstructured.UnstructuredList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message
E0105 11:53:39.853678 1315213 reflector.go:140] github.com/cloud/beirut-api/pkg/client/informers/externalversions/factory.go:116: Failed to watch *v1alpha1.DuplicateSet: failed to list *v1alpha1.DuplicateSet: object *unstructured.UnstructuredList does not implement the protobuf marshalling interface and cannot be encoded to a protobuf message

xigang avatar Jan 05 '24 03:01 xigang 

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

If you set AcceptContentTypes as application/vnd.kubernetes.protobuf,application/json, response type should be adaptive, pb for intree objects, and json for crs.

Have you actually encountered this problem？

@ikaven1024 That's right. If the protobuf content type is not supported, it fallback to json for crd. but object is *unstructured.UnstructuredList type will have this issue.

xigang avatar Jan 08 '24 11:01 xigang 

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

If you set AcceptContentTypes as application/vnd.kubernetes.protobuf,application/json, response type should be adaptive, pb for intree objects, and json for crs. Have you actually encountered this problem？

@ikaven1024 That's right. If the protobuf content type is not supported, it fallback to json for crd. but object is *unstructured.UnstructuredList type will have this issue.

It should be fixed, plz try it again.

ikaven1024 avatar Jan 08 '24 18:01 ikaven1024

@ikaven1024 Perfect, now all types of resources work well.

xigang avatar Jan 09 '24 02:01 xigang

i am using this draft and still having issues with unable to install api resources: unable to setup API, the details as below: resource="resourceRegistries.search.karmada.io" path="<storage-prefix>//search.karmada.io/resourceRegistries" I0402 14:03:27.739662 1 balancer.go:183] "[balancer] base.baseBalancer: handle SubConn state change: 0xc000010330, CONNECTING\n" I0402 14:03:27.739705 1 reflector.go:325] Listing and watching *search.ResourceRegistry from storage/cacher.go:/search.karmada.io/resourceRegistries I0402 14:03:27.740774 1 etcd.go:400] "Using watch cache" resource="resourceregistries.search.karmada.io" I0402 14:03:27.740891 1 etcd.go:400] "Using watch cache" resource="resourceregistries.search.karmada.io" E0402 14:03:27.741105 1 run.go:74] "command failed" err="unable to install api resources: unable to setup API &{[search.karmada.io/v1alpha1] map[v1alpha1:map[proxying:0xc000013288 resourceregistries:0xc000013268 resourceregistries/status:0xc000013280 search:0xc0012176b0]] v1 <nil> 0xc000983490 {0xc000983490 0xc000116c78 [{application/json application json true 0xc0002a1d10 0xc0002a1d60 0xc0002a1db0 0xc0004d92f0} {application/yaml application yaml true 0xc0002a1e00 <nil> 0xc0002a1e50 <nil>} {application/vnd.kubernetes.protobuf application vnd.kubernetes.protobuf false 0xc00034d200 <nil> 0xc00034d200 0xc0004d9380}] 0xc0002a1d10} 0xc00034d240 map[com.github.karmada-io.karmada.pkg.apis.cluster.v1alpha1.LocalSecretReference:0xc000cca240 com.github.karmada-io.karmada.pkg.apis.policy.v1alpha1.ClusterAffinity:0xc000b0bd40 com.github.karmada-io.karmada.pkg.apis.policy.v1alpha1.FieldSelector:0xc000b12d80 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.BackendStoreConfig:0xc000b7e240 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.OpenSearchConfig:0xc000cc98c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.Proxying:0xc000cfa480 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceRegistry:0xc0007566c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceRegistrySpec:0xc000b058c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceRegistryStatus:0xc000cd2240 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceSelector:0xc000ccb8c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.Search:0xc000cf8b40 io.k8s.api.core.v1.NodeSelectorRequirement:0xc000b13440 io.k8s.apimachinery.pkg.apis.meta.v1.Condition:0xc000cda000 io.k8s.apimachinery.pkg.apis.meta.v1.FieldsV1:0xc000cf4b40 io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector:0xc000b19200 io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelectorRequirement:0xc000b19b00 io.k8s.apimachinery.pkg.apis.meta.v1.ManagedFieldsEntry:0xc000cf06c0 io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta:0xc000cdf440 io.k8s.apimachinery.pkg.apis.meta.v1.OwnerReference:0xc000cf5440 io.k8s.apimachinery.pkg.apis.meta.v1.Time:0xc000cdc000]}: [error in registering resource: proxying, resource proxying must implement SingularNameProvider, error in registering resource: search, resource search must implement SingularNameProvider]"

wangxf1987 avatar Apr 02 '24 14:04 wangxf1987

i am using this draft and still having issues with unable to install api resources: unable to setup API, the details as below: I0402 14:03:27.739130 1 clientconn.go:188] "[core] [Channel #13] Channel created\n" I0402 14:03:27.739160 1 logging.go:43] "[core] [Channel #13] original dial target is: \"etcd-endpoints://0xc000992fc0/etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local:2379\"\n" I0402 14:03:27.739212 1 logging.go:43] "[core] [Channel #13] parsed dial target is: {Scheme:etcd-endpoints Authority:0xc000992fc0 URL:{Scheme:etcd-endpoints Opaque: User: Host:0xc000992fc0 Path:/etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local:2379 RawPath: OmitHost:false ForceQuery:false RawQuery: Fragment: RawFragment:}}\n" I0402 14:03:27.739229 1 logging.go:43] "[core] [Channel #13] Channel authority set to \"etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local:2379\"\n" I0402 14:03:27.739307 1 logging.go:43] "[core] [Channel #13] Resolver state updated: {\n \"Addresses\": [\n {\n \"Addr\": \"etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local:2379\",\n \"ServerName\": \"etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n }\n ],\n \"ServiceConfig\": {\n \"Config\": {\n \"Config\": null,\n \"LB\": \"round_robin\",\n \"Methods\": {}\n },\n \"Err\": null\n },\n \"Attributes\": null\n} (service config updated; resolver returned new addresses)\n" I0402 14:03:27.739333 1 logging.go:43] "[core] [Channel #13] Channel switches to new LB policy \"round_robin\"\n" I0402 14:03:27.739379 1 balancer.go:98] "[balancer] base.baseBalancer: got new ClientConn state: {{[{\n \"Addr\": \"etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local:2379\",\n \"ServerName\": \"etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local\",\n \"Attributes\": null,\n \"BalancerAttributes\": null,\n \"Type\": 0,\n \"Metadata\": null\n}] 0xc000209dc0 <nil>} <nil>}\n" I0402 14:03:27.739401 1 clientconn.go:722] "[core] [Channel #13 SubChannel #14] Subchannel created\n" I0402 14:03:27.739427 1 roundrobin.go:50] "[roundrobin] roundrobinPicker: Build called with info: {map[]}\n" I0402 14:03:27.739438 1 logging.go:43] "[core] [Channel #13] Channel Connectivity change to CONNECTING\n" I0402 14:03:27.739453 1 logging.go:43] "[core] [Channel #13 SubChannel #14] Subchannel Connectivity change to CONNECTING\n" I0402 14:03:27.739470 1 logging.go:43] "[core] [Channel #13 SubChannel #14] Subchannel picks a new address \"etcd-client.fedcluster-nj7h4xgklvnl.svc.cluster.local:2379\" to connect\n" I0402 14:03:27.739493 1 store.go:1579] "Monitoring resource count at path" resource="resourceRegistries.search.karmada.io" path="<storage-prefix>//search.karmada.io/resourceRegistries" I0402 14:03:27.739662 1 balancer.go:183] "[balancer] base.baseBalancer: handle SubConn state change: 0xc000010330, CONNECTING\n" I0402 14:03:27.739705 1 reflector.go:325] Listing and watching *search.ResourceRegistry from storage/cacher.go:/search.karmada.io/resourceRegistries I0402 14:03:27.740774 1 etcd.go:400] "Using watch cache" resource="resourceregistries.search.karmada.io" I0402 14:03:27.740891 1 etcd.go:400] "Using watch cache" resource="resourceregistries.search.karmada.io" E0402 14:03:27.741105 1 run.go:74] "command failed" err="unable to install api resources: unable to setup API &{[search.karmada.io/v1alpha1] map[v1alpha1:map[proxying:0xc000013288 resourceregistries:0xc000013268 resourceregistries/status:0xc000013280 search:0xc0012176b0]] v1 <nil> 0xc000983490 {0xc000983490 0xc000116c78 [{application/json application json true 0xc0002a1d10 0xc0002a1d60 0xc0002a1db0 0xc0004d92f0} {application/yaml application yaml true 0xc0002a1e00 <nil> 0xc0002a1e50 <nil>} {application/vnd.kubernetes.protobuf application vnd.kubernetes.protobuf false 0xc00034d200 <nil> 0xc00034d200 0xc0004d9380}] 0xc0002a1d10} 0xc00034d240 map[com.github.karmada-io.karmada.pkg.apis.cluster.v1alpha1.LocalSecretReference:0xc000cca240 com.github.karmada-io.karmada.pkg.apis.policy.v1alpha1.ClusterAffinity:0xc000b0bd40 com.github.karmada-io.karmada.pkg.apis.policy.v1alpha1.FieldSelector:0xc000b12d80 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.BackendStoreConfig:0xc000b7e240 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.OpenSearchConfig:0xc000cc98c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.Proxying:0xc000cfa480 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceRegistry:0xc0007566c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceRegistrySpec:0xc000b058c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceRegistryStatus:0xc000cd2240 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.ResourceSelector:0xc000ccb8c0 com.github.karmada-io.karmada.pkg.apis.search.v1alpha1.Search:0xc000cf8b40 io.k8s.api.core.v1.NodeSelectorRequirement:0xc000b13440 io.k8s.apimachinery.pkg.apis.meta.v1.Condition:0xc000cda000 io.k8s.apimachinery.pkg.apis.meta.v1.FieldsV1:0xc000cf4b40 io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelector:0xc000b19200 io.k8s.apimachinery.pkg.apis.meta.v1.LabelSelectorRequirement:0xc000b19b00 io.k8s.apimachinery.pkg.apis.meta.v1.ManagedFieldsEntry:0xc000cf06c0 io.k8s.apimachinery.pkg.apis.meta.v1.ObjectMeta:0xc000cdf440 io.k8s.apimachinery.pkg.apis.meta.v1.OwnerReference:0xc000cf5440 io.k8s.apimachinery.pkg.apis.meta.v1.Time:0xc000cdc000]}: [error in registering resource: proxying, resource proxying must implement SingularNameProvider, error in registering resource: search, resource search must implement SingularNameProvider]"

i am try to debug and print the store in the log, and i can see the value? Something probably went wrong

wangxf1987 avatar Apr 02 '24 14:04 wangxf1987

i am try to debug and print the store in the log, and i can see the value? Something probably went wrong

Would you like describe what do you request?

ikaven1024 avatar Apr 06 '24 13:04 ikaven1024

i am try to debug and print the store in the log, and i can see the value? Something probably went wrong

Would you like describe what do you request?

it's need to implement to GetSingularName func in the proxy.go and search.go, I was fixed and it's does work.

wangxf1987 avatar Apr 09 '24 07:04 wangxf1987

i am try to debug and print the store in the log, and i can see the value? Something probably went wrong

Would you like describe what do you request?

it's need to implement to GetSingularName func in the proxy.go and search.go, I was fixed and it's does work.

Would you commit pr for it?

ikaven1024 avatar Apr 09 '24 14:04 ikaven1024

Hi @xigang , I have changed to structed objects in cache, and you can request in pb type. Set it in your client's restconfig:

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

And you will check it with --v=8. e.g.:

I1216 00:10:29.331099    5097 round_trippers.go:469] Request Headers:
I1216 00:10:29.331104    5097 round_trippers.go:473]     Accept: application/vnd.kubernetes.protobuf,application/json
I1216 00:10:29.331108    5097 round_trippers.go:473]     User-Agent: ___1go_build_github_com_karmada_io_karmada__output/v0.0.0 (darwin/arm64) kubernetes/$Format
I1216 00:10:29.570068    5097 round_trippers.go:574] Response Status: 200 OK in 238 milliseconds
I1216 00:10:29.570096    5097 round_trippers.go:577] Response Headers:
I1216 00:10:29.570101    5097 round_trippers.go:580]     Audit-Id: 461a6a5d-09ae-4449-9021-9fc1d8cc3cb1
I1216 00:10:29.570111    5097 round_trippers.go:580]     Audit-Id: 461a6a5d-09ae-4449-9021-9fc1d8cc3cb1
I1216 00:10:29.570119    5097 round_trippers.go:580]     Cache-Control: no-cache, private
I1216 00:10:29.570122    5097 round_trippers.go:580]     Cache-Control: no-cache, private
I1216 00:10:29.570126    5097 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf

Would you test the performance with it again?

restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

If you set AcceptContentTypes as application/vnd.kubernetes.protobuf,application/json, response type should be adaptive, pb for intree objects, and json for crs.

Have you actually encountered this problem？

@ikaven1024 Hi, may i ask whick file this code should be modified in?

wangxf1987 avatar May 08 '24 02:05 wangxf1987

i am try to debug and print the store in the log, and i can see the value? Something probably went wrong

Would you like describe what do you request?

it's need to implement to GetSingularName func in the proxy.go and search.go, I was fixed and it's does work.

Would you commit pr for it?

@ikaven1024 It's has been fixed in the latest release

wangxf1987 avatar May 08 '24 02:05 wangxf1987

@ikaven1024 Hi, may i ask whick file this code should be modified in?

If you are using client-go requesting karmada-search, you can set it as below:

restConfig, err := clientcmd.BuildConfigFromFlags("", "")
restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

Full example:

func main() {
	fs := &flag.FlagSet{}
	klog.InitFlags(fs)
	_ = fs.Set("v", "8")

	home, _ := os.UserHomeDir()
	restConfig, _ := clientcmd.BuildConfigFromFlags("", filepath.Join(home, ".kube/config"))
	restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

	clientset := kubernetes.NewForConfigOrDie(restConfig)
	clientset.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
}

And you can see response's Content-Type in log:

I0508 17:00:45.931528   20709 round_trippers.go:577] Response Headers:
I0508 17:00:45.931551   20709 round_trippers.go:580]     Cache-Control: no-cache, private
I0508 17:00:45.931572   20709 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf

ikaven1024 avatar May 08 '24 09:05 ikaven1024

@ikaven1024 Hi, may i ask whick file this code should be modified in?

If you are using client-go requesting karmada-search, you can set it as below:

restConfig, err := clientcmd.BuildConfigFromFlags("", "")
restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

Full example:

func main() {
	fs := &flag.FlagSet{}
	klog.InitFlags(fs)
	_ = fs.Set("v", "8")

	home, _ := os.UserHomeDir()
	restConfig, _ := clientcmd.BuildConfigFromFlags("", filepath.Join(home, ".kube/config"))
	restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

	clientset := kubernetes.NewForConfigOrDie(restConfig)
	clientset.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
}

And you can see response's Content-Type in log:

I0508 17:00:45.931528   20709 round_trippers.go:577] Response Headers:
I0508 17:00:45.931551   20709 round_trippers.go:580]     Cache-Control: no-cache, private
I0508 17:00:45.931572   20709 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf

Thx, I try to modify member cluster client in pkg/util/membercluster_client.go#L212-L215, like this:

// BuildClusterConfig return rest config for member cluster.
func BuildClusterConfig(clusterName string,
       ...

        // modify here
	clusterConfig.AcceptContentTypes = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.ContentType = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.QPS = 200
	clusterConfig.Burst = 400

	return clusterConfig, nil
}

I think this will also help improve access to member-cluster resources, and our test crd resources can be supported.

wangxf1987 avatar May 09 '24 03:05 wangxf1987

@ikaven1024 Hi, may i ask whick file this code should be modified in?

If you are using client-go requesting karmada-search, you can set it as below:

restConfig, err := clientcmd.BuildConfigFromFlags("", "")
restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

Full example:

func main() {
	fs := &flag.FlagSet{}
	klog.InitFlags(fs)
	_ = fs.Set("v", "8")

	home, _ := os.UserHomeDir()
	restConfig, _ := clientcmd.BuildConfigFromFlags("", filepath.Join(home, ".kube/config"))
	restConfig.AcceptContentTypes = "application/vnd.kubernetes.protobuf,application/json"

	clientset := kubernetes.NewForConfigOrDie(restConfig)
	clientset.CoreV1().Nodes().List(context.Background(), metav1.ListOptions{})
}

And you can see response's Content-Type in log:

I0508 17:00:45.931528   20709 round_trippers.go:577] Response Headers:
I0508 17:00:45.931551   20709 round_trippers.go:580]     Cache-Control: no-cache, private
I0508 17:00:45.931572   20709 round_trippers.go:580]     Content-Type: application/vnd.kubernetes.protobuf

Thx, I try to modify member cluster client in pkg/util/membercluster_client.go#L212-L215, like this:

// BuildClusterConfig return rest config for member cluster.
func BuildClusterConfig(clusterName string,
       ...

        // modify here
	clusterConfig.AcceptContentTypes = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.ContentType = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.QPS = 200
	clusterConfig.Burst = 400

	return clusterConfig, nil
}

I think this will also help improve access to member-cluster resources, and our test crd resources can be supported.

About QPS and burst value, it should be setting by user.

wangxf1987 avatar May 09 '24 03:05 wangxf1987

Thx, I try to modify member cluster client in pkg/util/membercluster_client.go#L212-L215, like this:

// BuildClusterConfig return rest config for member cluster.
func BuildClusterConfig(clusterName string,
       ...

        // modify here
	clusterConfig.AcceptContentTypes = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.ContentType = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.QPS = 200
	clusterConfig.Burst = 400

	return clusterConfig, nil
}

I think this will also help improve access to member-cluster resources, and our test crd resources can be supported.

It's an excellent advise. Would you create another issue for this topic.

ikaven1024 avatar May 09 '24 03:05 ikaven1024

Thx, I try to modify member cluster client in pkg/util/membercluster_client.go#L212-L215, like this:

// BuildClusterConfig return rest config for member cluster.
func BuildClusterConfig(clusterName string,
       ...

        // modify here
	clusterConfig.AcceptContentTypes = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.ContentType = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.QPS = 200
	clusterConfig.Burst = 400

	return clusterConfig, nil
}

I think this will also help improve access to member-cluster resources, and our test crd resources can be supported.

It's an excellent advise. Would you create another issue for this topic.

ok, I will be commit it by a new PR.

wangxf1987 avatar May 09 '24 06:05 wangxf1987

Thx, I try to modify member cluster client in pkg/util/membercluster_client.go#L212-L215, like this:

// BuildClusterConfig return rest config for member cluster.
func BuildClusterConfig(clusterName string,
       ...

        // modify here
	clusterConfig.AcceptContentTypes = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.ContentType = strings.Join([]string{runtime.ContentTypeProtobuf, runtime.ContentTypeJSON}, ",")
	clusterConfig.QPS = 200
	clusterConfig.Burst = 400

	return clusterConfig, nil
}

I think this will also help improve access to member-cluster resources, and our test crd resources can be supported.

It's an excellent advise. Would you create another issue for this topic.

ok, I will be commit it by a new PR.

https://github.com/karmada-io/karmada/pull/4936

wangxf1987 avatar May 13 '24 07:05 wangxf1987