Pointing to authenticating proxies/ingress controllers

[dhess]

Hi, thanks for this fantastic operator!

In this comment, in response to a question about how to point a Cloudflare tunnel at an authenticating proxy/ingress controller, you referred to the cfargotunnel/target annotation (https://github.com/adyanth/cloudflare-operator/issues/28):

For ingress support, check this: https://github.com/adyanth/cloudflare-operator/issues/28. You can overwrite where cloudflared points to the ingress using cfargotunnel.com/target annotation. OAuth2 in front of the service and adding Traefik middlewares like redirection was the goal of this. https://github.com/adyanth/cloudflare-operator/blob/main/docs/configuration.md#service-annotations

However, unless I'm mistaken, that feature was removed when you simplified the operator's design in https://github.com/adyanth/cloudflare-operator/pull/63. If that feature has been replaced by something equivalent in the new design, I must be missing it, and it's not clear to me how we're meant to point the Cloudflare tunnel at an ingress controller anymore.

In my particular case, I'm trying to use vmauth from the VictoriaMetrics Operator to add some bearer token-authentication to an exposed VictoriaMetrics service:

https://docs.victoriametrics.com/operator/auth.html

But I can only figure out how to connect the Cloudflare tunnel directly to the VMAuth service, and not its ingress.

[adyanth]

Hey @dhess , that is correct, in the new version, you can point directly to the service rather than the ingress. In the traefik example, you can point it at the traefik's service and it will work the same. Does pointing to the service not work as expected? I am not sure how VM implements their auth, but at some level it would have to pass through a K8s service belonging to VMAuth to which you could point this operator to.

Regarding #28, you can still do that by using the subjects[].spec.target. Example would look like this for guacamole. The service should be a valid one, otherwise you will see warnings on the CRD, but even without it being valid or a dummy service like the k8s default, it should work.

subjects:
  - kind: Service
    name: guacamole
    spec:
      fqdn: guacamole.domain
      noTlsVerify: false
      target: https://guacamole.local # <- use this field here to point to VMAuth

If not, this would be a feature request to support Ingress, which is not that bad, but would not be of much use when you can point to a service/FQDN.

[dhess]

VMAuth is mostly working with the Cloudflare tunnel (pointing to the service, rather than the ingress), but it does expose the VMAuth proxy's /metrics endpoint to the tunnel, in addition to a few other sensitive endpoints. These can be protected behind a token, but I'd prefer not to expose them at all.

It's not clear to me whether the ingress would have the same behavior as the service, as I haven't been able to test it because I don't know how to point the tunnel at it. However, based on the documentation, I get the impression that it wouldn't, as you're supposed to be able to route /metrics to a running VMAgent instance, which doesn't work as expected when I try to access it through the Cloudflare tunnel — I always get the VMAuth /metrics instead:

https://docs.victoriametrics.com/operator/auth.html

Other endpoints that the VMAuth proxy lacks, such as /target or /loki, work as expected.

[adyanth]

Could you let me know which service do you point the operator to? What ingress controller do you use and how does it currently block access to /metrics?

Also, setting the target as shown in the example above to expose what you want in different domains, would that be an option?

[dhess]

I'm not using an ingress controller in the cluster, as I haven't needed one thanks to cloudflare-operator, but the VictoriaMetrics operator creates an ingress for the VMAuth service:

$ kubectl get -A ingress                                                                                                                    ~/git/hackworth-gitops
NAMESPACE    NAME                 CLASS   HOSTS   ADDRESS   PORTS   AGE
monitoring   vmauth-main-router   nginx   *                 80      3d10h

apiVersion: operator.victoriametrics.com/v1beta1
kind: VMAuth
metadata:
  name: main-router
spec:
  podMetadata:
    labels:
      component: vmauth
  userSelector: {}
  userNamespaceSelector: {}
  replicaCount: 2
  resources:
    requests:
      cpu: "250m"
      memory: "350Mi"
    limits:
     cpu: "500m"
     memory: "850Mi"
  ingress:
    class_name: nginx

(Not that the VictoriaMetrics Operator automatically appends vmauth- to the VMAuth's metadata.name.)

Here's the TunnelBinding:

apiVersion: networking.cfargotunnel.com/v1alpha1
kind: TunnelBinding
metadata:
  name: vm-tunnelbinding
subjects:
  - kind: Service
    name: vmauth-main-router
    spec:
      fqdn: x.y.z
tunnelRef:
  kind: Tunnel
  name: monitoring-tunnel

I guess you're suggesting that I change the name of the ingress that the VM operator creates, and point the tunnel binding at that? I suppose that might be possible in one of the VMAuth's ingress settings.

[adyanth]

Oh, looks like victoriaMetrics/your cluster has an nginx ingress controller. You could try pointing the operator to the nginx's service which should then go through Nginx, which should do everything you expect it to.

If you do not have an ingress controller at all, does the setup without the Cloudflare operator work as expected? Are you able to locally connect to the endpoints you need while not being able to connect to /metrics? If not, I am guessing VictoriaMetrics assumes that you have an ingress controller and generates an ingress with certain config which does what you need it to do. But without an actual ingress controller to enforce it, it would not work.

[dhess]

The ingress is set up by the VictoriaMetrics operator, and there is no other ingress controller in the cluster, so there's no corresponding service for it, either. The nginx class you see there comes from the class_name property in the VMAuth config shown above. I set that because it's what was in the example in the VM documentation.

I haven't tried without the Cloudflare operator, but I'll try to find some time to do that. Thanks for your help!

[adyanth]

Ahh, in that case, as I mentioned, VM expects you to bring your own ingress to have those features. Currently, the ingress is created, but not being served by anyone, which means even if the operator supports ingress, it would not work. So, if you install an ingress controller to serve the ingress, and point the operator to the ingress' service, it should all work out. Let me know if that does not happen.