isolateVirtualHostsBySslConfig changing behavior in hybridGateways validation

[edubonifs]

Gloo Edge Product

Enterprise

Gloo Edge Version

1.15.2

Kubernetes Version

1.24

Describe the bug

I am seeing a behaviour in gateway proxy validation. I have an hybridGateway with delegatedHttpGateways. However, I don't have any HttpGateway created, so the gateway is failing with "warning: \n hybrid gateway does not have any populated matched gateways"

This is expected as I am not defining any gateway. However, if I set isolateVirtualHostsBySslConfig: true, then the validation runs and the gateway shows as Accepted, even though I have a gateway which delegates to non existing gateway.

Expected Behavior

Behavior should be the same doesn't matter the value of isolateVirtualHostsBySslConfig.

Steps to reproduce the bug

For reproducing, just apply this settings:

apiVersion: gloo.solo.io/v1
kind: Settings
metadata:
  annotations:
    meta.helm.sh/release-name: gloo
    meta.helm.sh/release-namespace: gloo-system
  labels:
    app: gloo
    app.kubernetes.io/managed-by: Helm
    gloo: settings
  name: default
  namespace: gloo-system
spec:
  consoleOptions:
    apiExplorerEnabled: true
    readOnly: false
  discovery:
    fdsMode: WHITELIST
  discoveryNamespace: gloo-system
  extauth:
    extauthzServerRef:
      name: extauth
      namespace: gloo-system
    transportApiVersion: V3
    userIdHeader: x-user-id
  gateway:
    enableGatewayController: true
    isolateVirtualHostsBySslConfig: true
    readGatewaysFromAllNamespaces: true
    validation:
      allowWarnings: true
      alwaysAccept: true
      disableTransformationValidation: true
      proxyValidationServerAddr: gloo:9988
      serverEnabled: true
      validationServerGrpcMaxSizeBytes: 104857600
      warnRouteShortCircuiting: false
  gloo:
    disableKubernetesDestinations: false
    disableProxyGarbageCollection: false
    enableRestEds: false
    invalidConfigPolicy:
      invalidRouteResponseBody: Gloo Gateway has invalid configuration. Administrators
        should run `glooctl check` to find and fix config errors.
      invalidRouteResponseCode: 404
      replaceInvalidRoutes: false
    proxyDebugBindAddr: 0.0.0.0:9966
    regexMaxProgramSize: 1024
    restXdsBindAddr: 0.0.0.0:9976
    xdsBindAddr: 0.0.0.0:9977
  graphqlOptions:
    schemaChangeValidationOptions:
      rejectBreakingChanges: false
  ratelimitServer:
    rateLimitBeforeAuth: false
    ratelimitServerRef:
      name: rate-limit
      namespace: gloo-system
  refreshRate: 60s

And this gateway:

apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  labels:
    app: gloo
  name: gateway-proxy-ssl
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8443
  hybridGateway:
    delegatedHttpGateways:
      httpConnectionManagerSettings:
        acceptHttp10: false
        defaultHostForHttp10: ""
        delayedCloseTimeout: 1s
        enableTrailers: true
        idleTimeout: 300s
        maxHeadersCount: 100
        maxRequestHeadersKb: 60
        mergeSlashes: true
        normalizePath: true
        pathWithEscapedSlashesAction: 1
        preserveCaseHeaderKeyFormat: true
        requestHeadersTimeout: 10s
        requestTimeout: 180s
        serverHeaderTransformation: 2
        skipXffAppend: false
        useRemoteAddress: true
      selector:
        labels:
          hybrid-gateway: ssl
      sslConfig:
        alpnProtocols:
        - http/1.1
        disableTlsSessionResumption: false
        parameters:
          cipherSuites:
          - ECDHE-RSA-AES128-GCM-SHA256
          - ECDHE-RSA-AES256-GCM-SHA384
          - ECDHE-RSA-CHACHA20-POLY1305
          ecdhCurves:
          - P-256
          - P-384
          - P-521
          maximumProtocolVersion: 4
          minimumProtocolVersion: 3
  options:
    perConnectionBufferLimitBytes: 131072
  proxyNames:
  - gateway-proxy
  ssl: true
  useProxyProto: false

You will see that the gateway appears as ACCEPTED, even though we don't have a httpgateway created. Then, change isolateVirtualHostsBySslConfig to false, and check again the status of the gateway.

You will see that the status of the gateway shows the following:

status:
  statuses:
    gloo-system:
      reason: "warning: \n  hybrid gateway does not have any populated matched gateways"
      reportedBy: gloo
      state: Warning

Additional Environment Detail

No response

Additional Context

No response

[ashishb-solo]

Possibly related issues:

https://github.com/solo-io/gloo/pull/8461 https://github.com/solo-io/gloo/pull/8407

In short, we had some issues in how errors were being registered for some of our listeners on hybrid gateways (tcp listeners specifically, I think, but my memory is very faint). I wonder if the issues in those pull requests are affecting the behaviour here too

[davidjumani]

So after investigating, this error only occurs when no VS exist

kg apply -f- << EOF
apiVersion: gateway.solo.io/v1
kind: VirtualService
metadata:
  name: test-direct-response
  namespace: gloo-system
spec:
  virtualHost:
    domains:
      - 'foo'
    routes:
      - matchers:
         - prefix: /
        directResponseAction:
          status: 200
          body: "Hello, world!"

---
apiVersion: gateway.solo.io/v1
kind: Gateway
metadata:
  name: gateway-proxy-ssl
  namespace: gloo-system
spec:
  bindAddress: '::'
  bindPort: 8443
  hybridGateway:
    delegatedHttpGateways:
      selector:
        labels:
          hybrid-gateway: SSL
EOF

virtualservice.gateway.solo.io/test-direct-response created
Error from server: error when creating "vs.yaml": admission webhook "gloo.gloo-system.svc" denied the request: resource incompatible with current Gloo snapshot: [Validating *v1.Gateway failed: 1 error occurred:
	* Validating *v1.Gateway failed: validating *v1.Gateway name:"gateway-proxy-ssl" namespace:"gloo-system": 1 error occurred:
	* could not render proxy: 2 errors occurred:
	* invalid resource gloo-system.gateway-proxy-ssl
	* WARN: 
  [hybrid gateway does not have any populated matched gateways]

[davidjumani]

This will be fixed in v1.17.0