[Bug] ERROR: Failed to extract ServerMetadata from context

[madhathacker]

Is this a support request?

• [X] This is not a support request

Is there an existing issue for this?

• [X] I have searched the existing issues

Current Behavior

I am running Headscale in a docker container. For some reason after editing the config.yaml while Headscale was running, I start getting spammed with the error ERROR: Failed to extract ServerMetadata from context in the Docker logs. Restarting the service doesn't seem to stop this. I am also running headscale-ui and headscale-admin but I don't think they are the problem here.

Expected Behavior

Edit the config while the server is running.

Steps To Reproduce

1. Start Headscale container.
2. From docker host, open the config.yaml
3. Make a change and write it to the config file

Environment

- OS: Ubuntu 22.04.4 LTS
- Docker: Version 25.0.3, build 4debf41
- Headscale version: headscale/headscale:v0.23.0-alpha12

Runtime environment

• [X] Headscale is behind a (reverse) proxy
• [X] Headscale runs in a container

Anything else?

I was simply trying to add OIDC to headscale by editing the config.

[kradalby]

Can you please try the latest beta, and ensure that the CLI binary and the server binary is the same version?

[agarzon-orangerine]

I see the same error, Im using docker image: headscale/headscale:0.23.0-rc.1 (latest at this moment)

My compose is quite straight forward:

services:
  headscale:
    image: headscale/headscale:0.23.0-rc.1
    restart: unless-stopped
    container_name: headscale
    environment:
      - TZ=America/Toronto
    volumes:
      - ./conf:/etc/headscale
      - headscale_data:/var/lib/headscale
    command: serve
    labels:
      - traefik.enable=true
      - traefik.http.routers.${COMPOSE_PROJECT_NAME}_headscale.rule=Host(`${DOMAIN}`)
      - traefik.http.routers.${COMPOSE_PROJECT_NAME}_headscale.tls=true
      - traefik.http.services.${COMPOSE_PROJECT_NAME}_headscale.loadbalancer.server.port=8080
      
volumes:
  headscale_data:

networks:
  default:
    external: true
    name: proxy

[kradalby]

This sounds proxy related, I found this from the upstream gRPC project we use: https://github.com/grpc-ecosystem/grpc-gateway/issues/2445

I've tested with my own server, putting it behind a nginx proxy I was unable to reproduce.

Please be mindful that we do not have the capacity to test headscale behind the X x Y x Z combinations of proxies and configuration, which is why it is not supported. You can manage, but your miles might vary.

We did a dependency update to that library, and that might be related, but the update of dependencies is more important than novel proxy configurations.

[Typhon0]

Hi, I was experiencing the same problem with Traefik V2, and upgrading to Traefik V3 solved it.

- Headscale v0.23.0
- Traefik 3.4.1

[kradalby]

Sounds like this is out of our hands, but should be mentioned in an faq or common proxy issues.

[davidhan888-code]

Hi, I was experiencing the same problem with Traefik V2, and upgrading to Traefik V3 solved it.

- Headscale v0.23.0
- Traefik 3.4.1

Thanks for the hint. I still get the same error. Besides it is 3.1.4 rather than 3.4.1

[appleimperio]

I'm having the same issue with Nginx Proxy Manager and headscale 0.23.0

[skarpinis]

yes same issue with NPM

[appleimperio]

In my case the problem was with the integration with a webui. I changed to headscale-admin and the error disappear.

[kradalby]

I would like to emphasis that we will not be taking any action to resolve this, the error seems to be as a result of us updating a dependency, in this case gRPC related. Since it is not an alternative to change it, nor to not update them, there is little we can or will attempt doing.

We often underline that we do not support proxies, one of the reasons is this, it seems particularly tricky to get gRPC working correctly. In addition, Tailscale's protocol does not always play nice with proxies either (as per https://github.com/juanfont/headscale/issues/1468).

I will probably close it a bit, but need to think about how to resurface it so we dont get new issues created about the same error.

[skarpinis]

I will ignore no another option

[reey]

FYI: had the same issue with https://github.com/ifargle/headscale-webui and traefik, but no problems using: https://github.com/GoodiesHQ/headscale-admin together with traefik

[dreem2001]

It can be caused enabled legacy API <23 in settings of Headscale-admin.

[appleimperio]

@dreem2001 That was exactly my issue. I disable it and everything start working

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.