Improve ext-auth startup time with many non-functional passthrough authconfigs

[huzlak]

Gloo Edge Product

Enterprise

Gloo Edge Version

1.15.8

Is your feature request related to a problem? Please describe.

In case you have more than 2k grpc passthrough authconfigs(probably config per service) and the passthrough auth service scaled down/unreachable and restart the ext-auth pod, it takes too long to become ready because /healthcheck responds with 500 until all the configs are processed. The configs are processed serially, which equals to 2000*5s where 5s is connectTimeout for grpc passthrough.

Describe the solution you'd like

I'd like the configs to be processed in parallel.

Describe alternatives you've considered

Lower the connectTimeout which makes the process faster

Additional Context

Setup I used to replicate during investigation:

kubectl apply -f- <<EOF
apiVersion: v1
kind: Pod
metadata:
  labels:
    gloo: grpc-passthrough-stable
  name: grpc-passthrough-stable
spec:
  containers:
    - image: ghcr.io/huzlak/grpc-passthrough-example:append2
      imagePullPolicy: Always
      name: grpc-passthrough
  restartPolicy: Always
---
apiVersion: v1
kind: Service
metadata:
  labels:
    app: grpc-passthrough-stable
  name: grpc-passthrough-stable
spec:
  ports:
  - name: grpc
    port: 17772
    protocol: TCP
    targetPort: 17772
  selector:
    gloo: grpc-passthrough-stable
EOF
kubectl apply -f- <<EOF
apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: grpc-passthrough-auth-stable
  namespace: gloo-system
spec:
  configs:
  - passThroughAuth:
      grpc:
        address: grpc-passthrough-stable.default:17772
        connectionTimeout: 3s
EOF

### Create 100 vs with own authconfig
for vs in $(seq 1 100)
do 
echo $vs
kubectl apply -f- <<EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: vs-$vs
  namespace: gloo-system
spec:
  virtualHost:
    options:
      extauth:
        configRef:
          name: grpc-passthrough-auth-$vs
          namespace: gloo-system    
    domains:
    - 'httpbin-$vs.test'
    routes:
    - matchers:
       - prefix: /
      routeAction:
        single:
          upstream:
            name: static-upstream
            namespace: gloo-system	 
EOF
kubectl apply -f- <<EOF
apiVersion: enterprise.gloo.solo.io/v1
kind: AuthConfig
metadata:
  name: grpc-passthrough-auth-$vs
  namespace: gloo-system
spec:
  configs:
  - passThroughAuth:
      grpc:
        address: grpc-passthrough.default:17772
        connectionTimeout: 100ms
EOF
done

Additional Resources

See Zendesk ticket for detailed use-case description.

[huzlak]

Internal slack discussion https://solo-io-corp.slack.com/archives/C03GJDXLE00/p1706713492433739

[soloio-bot]

Zendesk ticket #3182 has been linked to this issue.