v0.17.0-alpha2 doesn't serve on 443

[papamoose]

headscale serve does not serve anything on port 443 in version v0.17.0-alpha2

ss -tpln output only shows port :80 being used.

LISTEN 0      4096               *:80              *:*    users:(("headscale",pid=2234,fd=10)) 

If you use v0.17.0-alpha1 everything seems to be working as expected. At least I was able to register a new node with out issue.

ss -tpln output when running v0.17.0-apha1

LISTEN 0      4096       127.0.0.1:9090       0.0.0.0:*    users:(("headscale",pid=1830,fd=11))     
LISTEN 0      4096               *:80               *:*    users:(("headscale",pid=1830,fd=14))     
LISTEN 0      4096               *:443              *:*    users:(("headscale",pid=1830,fd=10))     
LISTEN 0      4096               *:50443            *:*    users:(("headscale",pid=1830,fd=9)) 

Both server and client are Ubuntu 22.04.

I'm a new user of headscale so it's possible I don't have something configured properly. This seems unlikely though as v0.17.0-alpha1 works and alpha2 does not.

[juanfont]

Can you paste here your config?

[papamoose]

• I don't change the config between switching versions.
• I download the binary release for amd64 from the releases page.
• I grab the example config from the git repo and save as config.yaml and modify the following keys: server_url, tls_letsencrypt_hostname, acme_email

---
server_url: https://vpn2.example.com:443
listen_addr: 0.0.0.0:8080
metrics_listen_addr: 127.0.0.1:9090
grpc_listen_addr: 0.0.0.0:50443
grpc_allow_insecure: false
private_key_path: /var/lib/headscale/private.key
noise:
  private_key_path: /var/lib/headscale/noise_private.key
ip_prefixes:
  - fd7a:115c:a1e0::/48
  - 100.64.0.0/10
derp:
  server:
    enabled: false
    region_id: 999
    region_code: "headscale"
    region_name: "Headscale Embedded DERP"
    stun_listen_addr: "0.0.0.0:3478"
  urls:
    - https://controlplane.tailscale.com/derpmap/default
  paths: []
  auto_update_enabled: true
  update_frequency: 24h
disable_check_updates: false
ephemeral_node_inactivity_timeout: 30m
node_update_check_interval: 10s
db_type: sqlite3
db_path: /var/lib/headscale/db.sqlite
acme_url: https://acme-v02.api.letsencrypt.org/directory
acme_email: "[email protected]"
tls_letsencrypt_hostname: "vpn2.example.com"
tls_client_auth_mode: relaxed
tls_letsencrypt_cache_dir: /var/lib/headscale/cache
tls_letsencrypt_challenge_type: HTTP-01
tls_letsencrypt_listen: ":http"
tls_cert_path: ""
tls_key_path: ""
log_level: info
acl_policy_path: ""
dns_config:
  nameservers:
    - 1.1.1.1
  domains: []
  magic_dns: true
  base_domain: example.com
unix_socket: /var/run/headscale.sock
unix_socket_permission: "0770"
logtail:
  enabled: false
randomize_client_port: false

root@localhost:/etc/headscale# headscale serve
An updated version of Headscale has been found (0.17.0-alpha2 vs. your current v0.17.0-alpha1). Check it out https://github.com/juanfont/headscale/releases
2022-09-07T23:54:24Z INF No private key file at path, creating... path=/var/lib/headscale/private.key
2022-09-07T23:54:24Z INF No private key file at path, creating... path=/var/lib/headscale/noise_private.key
2022-09-07T23:54:24Z INF Enabling remote gRPC at 0.0.0.0:50443
2022-09-07T23:54:24Z INF listening and serving gRPC on: 0.0.0.0:50443
2022-09-07T23:54:24Z INF listening and serving HTTP on: 0.0.0.0:8080
2022-09-07T23:54:24Z INF listening and serving metrics on: 127.0.0.1:9090
2022-09-07T23:54:24Z INF Setting up a DERPMap update worker frequency=86400000

root@localhost:/etc/headscale# headscale version
v0.17.0-alpha2
root@localhost:/etc/headscale# rm -r /var/lib/headscale/* /var/run/headscale.sock
root@localhost:/etc/headscale# headscale serve
2022-09-07T23:58:06Z INF No private key file at path, creating... path=/var/lib/headscale/private.key
2022-09-07T23:58:06Z INF No private key file at path, creating... path=/var/lib/headscale/noise_private.key
2022-09-07T23:58:07Z INF Setting up a DERPMap update worker frequency=86400000

[statkashaman]

Hello, you can change tls_letsencrypt_challenge_type from HTTP-01 to TLS-ALPN-01, or use existing certificate with options tls_cert_path: "" and tls_key_path: "". It's a bug (

[papamoose]

I was able to successfully start headscale using v0.17.0-alpha2 following @statkashaman instructions.

1. Set tls_letsencrypt_challenge_type: TLS-ALPN-01.
2. headscale version then complained about listen_addr needing to be set to :443
3. listen_addr: 0.0.0.0:8080 -> listen_addr: 0.0.0.0:443

root@localhost:/etc/headscale# headscale version
v0.17.0-alpha2
root@localhost:~# headscale serve
2022-09-08T11:28:06Z INF Enabling remote gRPC at 0.0.0.0:50443
2022-09-08T11:28:06Z INF listening and serving gRPC on: 0.0.0.0:50443
2022-09-08T11:28:06Z INF listening and serving HTTP on: 0.0.0.0:443
2022-09-08T11:28:06Z INF listening and serving metrics on: 127.0.0.1:9090
2022-09-08T11:28:06Z INF Setting up a DERPMap update worker frequency=86400000

It looks like there is no service running port 80. I assume that has to do with using TLS-ALPN-01.

root@localhost:~# ss -tpln|grep headscale
LISTEN 0      4096       127.0.0.1:9090       0.0.0.0:*    users:(("headscale",pid=4623,fd=14))     
LISTEN 0      4096               *:443              *:*    users:(("headscale",pid=4623,fd=13))     
LISTEN 0      4096               *:50443            *:*    users:(("headscale",pid=4623,fd=12)) 

I was then able to register two machines using the tailscale client who can ping each other. :)

[db48x]

Good to know that there is a workaround; too bad I cannot put Headscale on 443 here.

[victorhooi]

I possibly seem to be hitting this issue:

I've noticed what appears to be a regression between 0.17-alpha1 and 0.17-alpha2/alpha3.

Specifically - with 0.17-alpha1 - headscale is listening on the specified ports in the configuration file (e.g. 8080, 50443 etc.) as expected:

victorhooi@headscale-test:~$ sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      475280/sshd: /usr/s
tcp        0      0 127.0.0.1:9090          0.0.0.0:*               LISTEN      721302/headscale
tcp6       0      0 :::80                   :::*                    LISTEN      721302/headscale
tcp6       0      0 :::8080                 :::*                    LISTEN      721302/headscale
tcp6       0      0 :::22                   :::*                    LISTEN      475280/sshd: /usr/s
tcp6       0      0 :::50443                :::*                    LISTEN      721302/headscale
udp        0      0 0.0.0.0:68              0.0.0.0:*                           383/dhclient
udp        0      0 127.0.0.1:323           0.0.0.0:*                           22697/chronyd
udp6       0      0 ::1:323                 :::*                                22697/chronyd
victorhooi@headscale-test:~$

Here is the startup console output as well:

An updated version of Headscale has been found (0.17.0-alpha3 vs. your current v0.17.0-alpha1). Check it out https://github.com/juanfont/headscale/releases
2022-09-22T07:40:15Z INF Setting up a DERPMap update worker frequency=86400000
2022-09-22T07:40:15Z INF Enabling remote gRPC at 0.0.0.0:50443
2022-09-22T07:40:15Z INF listening and serving gRPC on: 0.0.0.0:50443
2022-09-22T07:40:15Z INF listening and serving HTTP on: 0.0.0.0:8080
2022-09-22T07:40:15Z INF listening and serving metrics on: 127.0.0.1:9090
2022-09-22T07:40:18Z INF The HTTP2 server was closed error=EOF
2022-09-22T07:40:18Z INF ../runner/work/headscale/headscale/protocol_common.go:677 > Machine successfully refreshed machine=Victors-MacBook-Pro-2 node_key=[bQLA7] noise=true old_node_key=[PgiEl]
2022-09-22T07:40:18Z INF Client is starting up. Probably interested in a DERP map handler=PollNetMap machine=Victors-MacBook-Pro-3 noise=true
2022-09-22T07:40:18Z INF Client is starting up. Probably interested in a DERP map handler=PollNetMap machine=Victors-MacBook-Pro-3 noise=true

However, with 0.17-alpha2 and 0.17-alpha3 and the same configuration file - it does not appear to be listening on those ports anymore:

victorhooi@headscale-test:~$ sudo netstat -tunlp
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      475280/sshd: /usr/s
tcp6       0      0 :::80                   :::*                    LISTEN      721251/headscale
tcp6       0      0 :::22                   :::*                    LISTEN      475280/sshd: /usr/s
udp        0      0 0.0.0.0:68              0.0.0.0:*                           383/dhclient
udp        0      0 127.0.0.1:323           0.0.0.0:*                           22697/chronyd
udp6       0      0 ::1:323                 :::*                                22697/chronyd

and the startup console lines:

victorhooi@headscale-test:~$ sudo headscale serve
2022-09-22T07:28:51Z DBG Loading DERPMap from path func=GetDERPMap url=https://controlplane.tailscale.com/derpmap/default
2022-09-22T07:28:51Z INF Setting up a DERPMap update worker frequency=86400000

I'm just trying to understand the workaround - does this mean we need to change the configured port from 8080 to 443 as well?

[kradalby]

I think this was a result of a refactoring, can some of you try https://github.com/juanfont/headscale/pull/831.

[kradalby]

This fix is now in 0.17 alpha 4, please give it a go.