Headscale fails to activate clients with postgresql backend

[ishanjain28]

Bug description

Tailscale clients authenticate successfully with headscale when headscale is configured to use postgres but then get stuck in a loop and keep refreshing keys.

More specifically,

In case of sqlite,

1. Clients sends 556byte payload, headscale answers it.
2. Then, Client sends ~629 byte payload containing the peerapi4, peerapi6 services and client responds
3. Then, client sends a 1000+ byte payload with read_only=false containing it's endpoints and after this everything works!

In case of postgres,

1. Client sends 556byte payload, headscale returns the exact same answer as sqlite.
2. Client sends 629byte payload(read only=true), headscale responds but KeyExpiry is set to 0001-01-01 05:53:28+05:53:28.
3. Client again sends a 629byte payload and this keeps happening in a loop. Client never sends a payload with read_only=false.

To Reproduce

1. Install postgresql 14
2. Install latest build of headscale(from git head or the latest release)
3. Configure it to use postgresql and run it

Try to register any tailscale client.

Context info

[kradalby]

Note for when this is tackled,

• We should first implement integration tests that shows this, then fix it to prevent regression
• We need to get rid of all timestamps that is not initialised/set (no more nil or 0001-01-01)

[QZAiXH]

I seem to have encountered this problem, the client cannot join headscale with authkey, and the tailscale status keeps showing Logged out

[QZAiXH]

Can you guys tell me how to solve this problem? Maybe I can x change the code accordingly. @kradalby @juanfont

[Yxnt]

Can you guys tell me how to solve this problem? Maybe I can x change the code accordingly. @kradalby @juanfont

@QZAiXH maybe i can solve this problem, but i need more time to test this case. #765 has already been fixed for the situation where the command line gets stuck when using tailscale up and tailscale login, but there are still more cases that need to be tested.

[github-actions[bot]]

This issue is stale because it has been open for 180 days with no activity.

[github-actions[bot]]

This issue was closed because it has been inactive for 14 days since being marked as stale.

[weikinhuang]

I'm also encountering this issue, clients status:

tailscale status
Logged out.

Even though I'm connected and have an ip. I'm using a reusable key (for a router device).

[TnZzZHlp]

I also encountered this problem. Headscale version: v0.23.0-alpha2 I use nginx for reverse proxy, I don't know if it is because of the problem caused by nginx

[weikinhuang]

Like the reporter, it works fine with sqlite. I was trying to move to postgres for HA, but encountered this issue and went back to sqlite.

[2kvasnikov]

Same issue with OpenWrt router as a tailscale client. Can register cleent with sqllite.

[alexfornuto]

Hate to leave a "bump" comment, but FYI this issue occurs on Postgres 15 as well.

[m1cr0man]

Can confirm this happens on Postgres 16 also. I have collected logs from headscale whilst trying to join with a reusable preauth key (log level debug). Also in the gist is the client status json output, and the server node info json output, after the registration. I redacted/obfuscated some data.

Clientside interaction looks like so:

$ sudo tailscale up --reset --login-server https://headscale.example.com --timeout 20s --authkey xyz123
timeout waiting for Tailscale service to enter a Running state; check health with "tailscale status"
$ sudo tailscale status
Logged out.
Log in at: https://headscale.example.com/register/nodekey:abc123

Can confirm that switching to SQLite resolves the issue. Perhaps it is a collation issue, wherein some comparison is returning different results depending on the engine? Happy to do more testing.

[sjansen1]

I always wonder why authkey is not working on my Headscale installation until i found this issue. For now, i auth with openid, move this nodes to a fake user that can not login and set expiration date in the database to a high value to avoid expiration. For me, thats the only way to avoid expiration on server/subnet gateways.