Serving Operator should provide an option that all ingress gateways are configured

[trshafer]

If a user creates a custom ingress gateway, istio may select a gateway at random (https://github.com/knative/serving/issues/5163#issuecomment-521815915). The operator should provide a way to ensure that the ingress gateway always works.

This relies a bit on https://github.com/knative/serving/issues/5172.

I believe this could work by montoring gateways and adjusting the config-istio to account for the new gateway (https://github.com/knative/serving/blob/b7dfe3fdb80ffac7f7a05c20787400ce51aad2b1/config/config-istio.yaml#L45)

[trshafer]

Another option is to not install the knative-ingress-gateway (https://github.com/knative/serving/blob/b7dfe3fdb80ffac7f7a05c20787400ce51aad2b1/config/202-gateway.yaml#L19) if one is already configured, . And use that gateway instead of the knative-ingress-gateway.

[zhruan]

/assign zhruan

[houshengbo]

I would say we are seeking for the support of multiple ingresses, instead of the only istio.

[houshengbo]

Some discussions in the community:

Three options I could see would be:
* Package all the plugins and have something in the CR to list which plugin dukes should be included
* Package all the plugins in one fine and use labels to select them
* Package each in their own operator with some way for the serving operator to ensure at least one is installed
11:23
Serving uses annotations and a default config to determine which plugins to use for HTTP routing (and they aren't mutually exclusive)

markusthoemmes:redhatnew:  11:26 AM
we could key off of the setting @argent is referring to, to pick one of the ingresses, but each potentially comes with its own set of configuration.
So maybe we have something like this?
type IngressConfiguration struct {
  Istio *IstioConfiguration
  Contour *ContourConfiguration
  Kourier *KourierConfiguration
}
And for each non-nil configuration, we install the relevant package?
11:26
IIRC there's precedence in K8s for this

Jim Crossley:redhat:  11:27 AM
yeah, i like that, and from the install docs, it looks like it's easy enough to grab the istio, kourier, etc yaml's from somewhere, i.e. we won't have to create them specifically for the operator