Support Annotations on VirtualGateways to use with External-DNS please.

[caleygoff-invitae]

Is your feature request related to a problem? Please describe. With the new 1.0 release and introduction to VirtualGateways. Its been asked that we pivot from the Istio primitives in which we were using for multiple reasons.

1. The istio Gateway object supports kubernetes native annotations which we use.
2. It allowed us to setup a simple TLS ingress object so that our service operators would not have to worry about simple TLS on their ingress objects, and each request coming into the cluster could use this ingress for 80/443 connections for all apps on the cluster.

For #2 above, we know we can use the default simple tls similarly with VirtualGateways provided by GlooMesh, its just that until the annotation thing is addressed it makes that a little harder to reach for... perhaps not, but we are aware that the default simple-tls can be done with the vgs currently Describe the solution you'd like If we are going to use VirtualGateways going forward, and the VirtualServices that are spun off of those objects by gloo-mesh. Then it is absolutely important that objects support Kubernetes annotations that can be specified by the Service Mesh administrator.

Specifically we need to be able to programatically update DNS entries created by the hosts entry from a given VirtualService which is associated with the Gateway object, that is sitting behind our NortPort ingress object.

Our Ingress NodePort object has an annotation on it called external-dns.alpha.kubernetes.io/hostname and before 1.0 we had our default Gateway object with the following annotation on it external-dns.alpha.kubernetes.io/target. Both of these annotations had the value of ${istio_internal_nlb_dns_name} which is eventually expanded to an AWS Network Loadbalancer CNAME address that we as administrators could never possibly know without looking.

We deploy clusters at scale and each cluster gets a single AWS Network Loadbalancer with its unique CNAME. Its pertinent this works for us, as we use this CNAME to create DNS records in AWS Route53 (and optionally CloudMap) so that we can easily point FQDN urls to the correct cluster for connections from external requests off the mesh/cluster.

You can read more on how this works with External-DNS here https://github.com/kubernetes-sigs/external-dns/blob/master/docs/faq.md#are-other-ingress-controllers-supported

You can read more on how External-DNS can be configured with Istio for Gateways and VirtualServices here https://github.com/kubernetes-sigs/external-dns/blob/master/docs/tutorials/istio.md

Describe alternatives you've considered Presently Gloo-mesh offers no other alternatives to make this work. We already hashed out automation around this using open source tooling, and this new gloo-mesh feature breaks it by following the advice of solo.io field techs and replacing our current istio gateways/virtualservice with the Gateway provided by Gloo-mesh.

I have tried putting both the annotations on the Service object that is the NodePort but this didn't work for us. Please advise as this is and will be a blocker when this is rolled out.

Additional context Add any other context or screenshots about the feature request here.

[chrisgaun]

Thanks for submitting. Will discuss with team today

[asayah]

@chrisgaun this is needed, to be able to use externaldns with NodePort Services

[asayah]

@caleygoff-invitae can you please post some examples of how you use to do it without Gloo Mesh (annotation on svc + annotation on vs), thanks

[caleygoff-invitae]

sure

This is the Gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/target: "${istio_internal_nlb_dns_name}"    👈 
  name: default-http-https-ingressgateway
  namespace: istio-gateway-ingress
spec:
  selector:
    istio: ingressgateway
    app: istio-ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
      tls:
        httpsRedirect: true
    - hosts:
        - '*'
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        mode: SIMPLE
        credentialName: cluster-internal-wildcard-cert

And this is the VirtualService

apiVersion: v1
kind: Service
metadata:
  annotations:
    external-dns.alpha.kubernetes.io/hostname: "${istio_internal_nlb_dns_name}"    👈 
  name: istio-ingressgateway-entrypoint
  namespace: istio-gateway-ingress
  labels:
    app: istio-ingressgateway
    istio: ingressgateway
spec:
  type: NodePort
  selector:
    istio: ingressgateway
    app: istio-ingressgateway
    istio.io/rev: default

  ports:
    - name: status-port
      nodePort: 32020
      port: 15020
      protocol: TCP
      targetPort: 15020
    - name: healthz
      nodePort: 32021
      port: 15021
      protocol: TCP
      targetPort: 15021
    - name: http2
      nodePort: 30080
      port: 80
      protocol: TCP
      targetPort: 8080
    - name: https
      nodePort: 30443
      port: 443
      protocol: TCP
      targetPort: 8443
    - name: tls
      nodePort: 32443
      port: 15443
      protocol: TCP
      targetPort: 15443

The above annotations will allow for us to use those as default for our developers in a given cluster and abstract away the tls bits and the CNAME bits associated with the elb. So a developer could come along in their respective namespaces and add a VirtualService with a host field that is a viable fqdn like below

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: bookinfo
spec:
  hosts:
    - "book-store.mesh.dev.specialcorp.net"    👈 
  gateways:
    - istio-system/simple-http-https-gateway
  http:
    - match:
        - uri:
            exact: /productpage
        - uri:
            prefix: /static
        - uri:
            exact: /login
        - uri:
            exact: /logout
        - uri:
            prefix: /api/v1/products
      route:
        - destination:
            host: productpage
            port:
              number: 9080
    - match:
        - uri:
            prefix: /ratings
      route:
        - destination:
            host: ratings
            port:
              number: 9080

and because the developer used our default gateway for their VirtualService eg: gateways: istio-system/simple-http-https-gateway ExternalDNS will see this association and notice the annotations on the default VirtualService and Gateway and if the ${istio_internal_nlb_dns_name}" variable was perhaps a CNAME: some-random-129e871298-cname.corp.net. Then if wired up to do so, ExternalDNS will insert a new record in our DNS registrar wherever and point the host on the developers VirtualService eg hosts: book-store.mesh.dev.specialcorp.net to some-random-129e871298-cname.corp.net so that it routes to the correct nlb and eventually to the correct clusters

DNS somewhere is update like below 👍

DOMAIN	CNAME
book-store.mesh.dev.specialcorp.net	some-random-129e871298-cname.corp.net