CA Certificates missing

[sunny-logic]

Bug description

Does the "v0.23.0-alpha5-debug" have the ca-certificates added because I see the below error which was reported in https://github.com/juanfont/headscale/issues/1462 and I believe was fixed in https://github.com/juanfont/headscale/pull/1463.

ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default

I can't use v0.22.3 because it's doesn't have an ARMv7 variant.

Environment

• Version of headscale used - "v0.23.0-alpha5-debug"
• OS (e.g. Linux, Mac, Cygwin, WSL, etc.) and version - Linux 5.10.103-v7+ #1529 SMP Tue Mar 8 12:21:37 GMT 2022 armv7l GNU/Linux
• Log output ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default

• [x] Headscale is behind a (reverse) proxy
• [x] Headscale runs in a container

To Reproduce

Use "v0.23.0-alpha5-debug"

Logs and attachments

[ohdearaugustin]

Does this problem appear with the let's encrypt certificate or your own certificate?

The container are not build with the Dockerfile anymore. We use the buildin from goreleaser

[sunny-logic]

@ohdearaugustin -Thanks for the reply. Yes, I am using let's encrypt certificate with Headscale docker.

[vvirtues]

I also have this error on a fresh install. I pulled from docker hub and have default config.yaml. I don't know what certificate im using but I changed the URL to 0.0.0.0:8080:8080 for external use

Logs:

2024-04-25 17:09:08 2024-04-25T21:09:08Z TRC DNS configuration loaded dns_config={"Nameservers":["1.1.1.1"],"Proxied":true,"Resolvers":[{"Addr":"1.1.1.1"}]}
2024-04-25 17:09:08 2024-04-25T21:09:08Z INF Opening database database=sqlite3 path=/var/lib/headscale/db.sqlite
2024-04-25 17:09:08 2024-04-25T21:09:08Z ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default
2024-04-25 17:09:08 2024-04-25T21:09:08Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region
2024-04-25 17:09:08 2024-04-25T21:09:08Z INF Setting up a DERPMap update worker frequency=86400000
2024-04-25 17:09:08 2024-04-25T21:09:08Z FTL home/runner/work/headscale/headscale/cmd/headscale/cli/server.go:26 > Error starting server error="initial DERPMap is empty, Headscale requires at least one entry"

[vvirtues]

Should I manually import the derp map locally as a temporary fix?

[vvirtues]

I also realized that I might be the extra backslash added at the end of the URL

[sunny-logic]

@ohdearaugustin - Sorry, It's not clear from your previous comment if this issue would this be fixed ?

The container are not build with the Dockerfile anymore. We use the buildin from goreleaser

[vvirtues]

do you mean https://github.com/juanfont/headscale/pkgs/container/headscale ?

[vvirtues]

its updated the same as the docker hub image so guess not

[sunny-logic]

I can't use v0.22.3 because it's doesn't have an ARMv7 variant. I cant use "v0.23.0-alpha5-debug" because it doesn't have the ca-certificates.

Are there any other options that would support ARMv7 ?

[vvirtues]

oh is it the debug versions that don't have them?

[vvirtues]

ok that solves my problem

[sunny-logic]

I've tried today to build a an image for the verion v0.22.3 because it's doesn't have an ARMv7 variant. I've used the Source code zip but encounter a certificate issue.

`=> [internal] load build context 0.4s => => transferring context: 9.81kB 0.3s => ERROR [stage-1 2/4] RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lis 6.7s

[stage-1 2/4] RUN apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* && apt-get clean: 2.600 Get:1 http://deb.debian.org/debian bullseye InRelease [116 kB] 2.809 Get:2 http://deb.debian.org/debian-security bullseye-security InRelease [48.4 kB] 2.809 Get:3 http://deb.debian.org/debian bullseye-updates InRelease [44.1 kB] 3.461 Err:1 http://deb.debian.org/debian bullseye InRelease 3.461 At least one invalid signature was encountered. 3.989 Err:2 http://deb.debian.org/debian-security bullseye-security InRelease 3.989 At least one invalid signature was encountered. 4.578 Err:3 http://deb.debian.org/debian bullseye-updates InRelease 4.578 At least one invalid signature was encountered. 4.609 Reading package lists... 4.738 W: GPG error: http://deb.debian.org/debian bullseye InRelease: At least one invalid signature was encountered. 4.738 E: The repository 'http://deb.debian.org/debian bullseye InRelease' is not signed. 4.738 W: GPG error: http://deb.debian.org/debian-security bullseye-security InRelease: At least one invalid signature was encountered. 4.738 E: The repository 'http://deb.debian.org/debian-security bullseye-security InRelease' is not signed. 4.738 W: GPG error: http://deb.debian.org/debian bullseye-updates InRelease: At least one invalid signature was encountered. 4.738 E: The repository 'http://deb.debian.org/debian bullseye-updates InRelease' is not signed. Dockerfile:19 18 | 19 | >>> RUN apt-get update
20 | >>> && apt-get install -y ca-certificates
21 | >>> && rm -rf /var/lib/apt/lists/*
22 | >>> && apt-get clean 23 | ERROR: failed to solve: process "/bin/sh -c apt-get update && apt-get install -y ca-certificates && rm -rf /var/lib/apt/lists/* && apt-get clean" did not complete successfully: exit code: 100 `

[tuxpeople]

I can confirm the issue. When using docker.io/headscale/headscale:0.23.0-alpha10-debug with default config, Headscale can not download the derp map from Tailscale:

headscale     | 2024-05-15T09:50:31Z ERR Could not load DERP map from path error="Get \"https://controlplane.tailscale.com/derpmap/default\": tls: failed to verify certificate: x509: certificate signed by unknown authority" func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default
headscale     | 2024-05-15T09:50:31Z WRN DERP map is empty, not a single DERP map datasource was loaded correctly or contained a region

When building my own version like the following, it works:

FROM docker.io/headscale/headscale:0.23.0-alpha10-debug
RUN apt-get update \
  && apt-get install --no-install-recommends --yes ca-certificates \
  && rm -rf /var/lib/apt/lists/* \
  && apt-get clean

[ohdearaugustin]

So just to summarize only the debug image has the problem with the CA.

The production image is working?

[ohdearaugustin]

So I took a look at it.

The main problem is that the currently used basic container used for building the debug image:

https://github.com/juanfont/headscale/blob/5ad0aa44cb3caa3d76ac3bfa4469c7f84a813f55/.goreleaser.yml#L161-L165

We are using a plain debian:12 docker image as base. This image doesn't include the ca-certificates. As we changed our container build pipeline to ko, which is included in goreleaser. We can't directly modify the base-images, without building an extra base image. This is definitely a drawback about using ko.

Therefore my solution is to change the base image also to distroless, as it includes the CAs per default and we won't need any changes. See PR #1950

Furthermore we also should change the Dockerfile.debug to be closer to real build process with ko. This will be a separate issue.