helpers.MustMultiNamespacedRouteTableClient is very slow with a large list of namespaces

[ccding]

Gloo Edge Product

Open Source

Gloo Edge Version

v1.15.5

Kubernetes Version

v1.26.6

Describe the bug

We are handling thousands of namespaces. It appears the MustMultiNamespacedRouteTableClient function does the following

for _, ns := range namespaceWhitelist {
	// verify the crd is registered and we have List permission
	// for the given namespaces
	if _, err := crdClient.ResourcesV1().Resources(ns).List(ctx, metav1.ListOptions{}); err != nil {
		return nil, errors.Wrapf(err, "list check failed")
	}
}

which is extremely slow.

I wonder

1. can we add a flag to bypass the check of each namespace?
2. if we simply pass in a namespaceWhitelist = []string{metav1.NamespaceAll}, will there be any side effect? I am not sure if gloo will unnecessarily watch all namespaces or do something based on namespaceWhitelist. For context, we have 1 million namespaces and wanted to shard them to 100 clients, each working on 10K of the namespaces

Expected Behavior

MustMultiNamespacedRouteTableClient function should return immediately

Steps to reproduce the bug

just call helpers.MustMultiNamespacedRouteTableClient with 1000 namespaces and measure how long it takes

Additional Environment Detail

No response

Additional Context

No response