[enhancement] Add the ability to swap out certificate resolver

[iameskild]

Description

Add the ability to swap out certificate resolver from TLS challenge to DNS.

Value/benefit

This would allow local deployments (with a domain-name/DNS records) to have signed certs. This would also provide a lot more flexibility in general.

[iameskild]

I have made a few passes at this (see branch dnsresolver) with no success yet using this guide for reference: https://doc.traefik.io/traefik/https/acme/#dnschallenge

@costrouc I have also tried creating a new, more broadly scoped Cloudflare API token and adding the associated ACME email address.

The logs from the qhub-traefik-ingress pods complain about permission denied for the acme.json:

time="2022-01-26T05:47:50Z" level=info msg="Configuration loaded from flags."
time="2022-01-26T05:47:50Z" level=error msg="The ACME resolver \"default\" is skipped from the resolvers list because: unable to get ACME account: open acme.json: permission denied"
W0126 05:47:51.103648       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
time="2022-01-26T05:47:51Z" level=error msg="the router dev-dask-gateway-561ec817f88af4a74a45@kubernetescrd uses a non-existent resolver: default"
time="2022-01-26T05:47:51Z" level=error msg="the router dev-jupyterhub-1aaa0a759c4852d07f24@kubernetescrd uses a non-existent resolver: default"
time="2022-01-26T05:47:51Z" level=error msg="the router dev-forwardauth-eb5d793ddccc8fd18b4d@kubernetescrd uses a non-existent resolver: default"
time="2022-01-26T05:47:51Z" level=error msg="the router dev-grafana-ingress-route-95ed2e2c625ca4d66e53@kubernetescrd uses a non-existent resolver: default"
W0126 05:47:52.040382       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress
W0126 05:47:52.174895       1 warnings.go:70] networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList
W0126 05:47:52.220463       1 warnings.go:70] networking.k8s.io/v1beta1 IngressClass is deprecated in v1.19+, unavailable in v1.22+; use networking.k8s.io/v1 IngressClassList
time="2022-01-26T05:47:52Z" level=error msg="the router dev-grafana-ingress-route-95ed2e2c625ca4d66e53@kubernetescrd uses a non-existent resolver: default"
time="2022-01-26T05:47:52Z" level=error msg="the router dev-jupyterhub-1aaa0a759c4852d07f24@kubernetescrd uses a non-existent resolver: default"
time="2022-01-26T05:47:52Z" level=error msg="the router dev-dask-gateway-561ec817f88af4a74a45@kubernetescrd uses a non-existent resolver: default"
time="2022-01-26T05:47:52Z" level=error msg="the router dev-forwardauth-eb5d793ddccc8fd18b4d@kubernetescrd uses a non-existent resolver: default"
W0126 05:53:11.044068       1 warnings.go:70] extensions/v1beta1 Ingress is deprecated in v1.14+, unavailable in v1.22+; use networking.k8s.io/v1 Ingress

[costrouc]

I'd like to make this a high priority for the qhub 0.4.2. This will allow us a way to have secure valid certs for qhub local deployments and allow getting certificates for clusters that are not exposed to the internet. Enterprise developments will find this important to have.

[iameskild]

@costrouc I moved this issue to a future milestone partly so that we can focus on a smaller subset of issues/enhancements for the 0.4.2 release and partly because I have't been able to figure out how this would work yet.