juanfont
[Bug] Internal server error when logging in using Google OIDC
Is this a support request?
- [x] This is not a support request
Is there an existing issue for this?
- [x] I have searched the existing issues
Current Behavior
Similar to https://github.com/juanfont/headscale/issues/2465 which was closed.
An existing (since Headscale 0.23) tried to log in for the first time since upgrading to 0.25, without success.
2025-03-13T08:09:14Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-13T08:09:24Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"Bad Request\"" code=403
2025-03-13T08:09:34Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-13T08:10:07Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-13T08:11:07Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
Even after destroying the user (along with the associated nodes and preauthkeys) and restarting headscale, the issue persists.
I tried to investigate how the OIDC handling has changed over the versions, but from what I could gather most issues seem to stem from the email having possibly changed? This has not happened in our case.
Expected Behavior
The current user should be migrated to the new format.
Steps To Reproduce
- Setup a user with Google OIDC on Headscale 0.23.0.
- Migrate to 0.24 and then 0.25
- Try to reauthenticate
Environment
- OS: macOS 15.3.1
- Headscale version: 0.25.1
- Tailscale version: 1.80.2
Runtime environment
- [x] Headscale is behind a (reverse) proxy
- [x] Headscale runs in a container
Anything else?
# config.yaml
map_legacy_users: true
oidc:
only_start_if_oidc_is_available: true
issuer: https://accounts.google.com
client_id: <redacted>
client_secret_path: /secrets/headscale/oidc_client_secret
expiry: 180d
use_expiry_from_token: false
scope: ["openid", "profile", "email"]
extra_params:
domain_hint: <redacted>
allowed_domains:
- <redacted>
strip_email_domain: true
Thank you for your dedicated work on Headscale!
Can you provide a list of users in the database and their fields and the ones that try to login when you get the error?
I suspect that they might not have migrated from the old "user only" format to the new one, see the changelog for 0.24.
Can you provide a list of users in the database and their fields and the ones that try to login when you get the error?
I suspect that they might not have migrated from the old "user only" format to the new one, see the changelog for 0.24.
Sure, here you go! User 1 is our subnet router configured as a non-OIDC user. User 15 is an ordinary OIDC user, just with a special email.
sqlite> select * from users;
id | created_at | updated_at | deleted_at | name | display_name | email | provider_identifier | provider | profile_pic_url
1 | 2024-03-08 22:00:05.052505772+00:00 | 2024-03-08 22:00:05.052505772+00:00 | | pod-node | | | | |
2 | 2024-03-08 22:01:43.602676781+00:00 | 2025-03-13 10:04:41.871021878+00:00 | | adrian.goransson | Adrian Göransson | adrian.goransson@<domain> | <id_1> | oidc | <url_2>
4 | 2024-03-14 13:19:43.923617972+00:00 | 2024-03-14 13:19:43.923617972+00:00 | | <first>.<last> | | | | |
5 | 2024-03-15 15:43:18.872487023+00:00 | 2024-03-15 15:43:18.872487023+00:00 | | <first>.<last> | | | | |
7 | 2024-05-30 07:32:14.825899374+00:00 | 2024-05-30 07:32:14.825899374+00:00 | | <first>.<last> | | | | |
8 | 2024-07-15 11:11:48.313729614+00:00 | 2024-07-15 11:11:48.313729614+00:00 | | <first>.<last> | | | | |
9 | 2024-08-26 07:41:05.647379266+00:00 | 2024-08-26 07:41:05.647379266+00:00 | | <first>.<last> | | | | |
10 | 2024-08-27 09:37:05.68427993+00:00 | 2024-08-27 09:37:05.68427993+00:00 | | <first>.<last> | | | | |
11 | 2024-09-25 08:05:59.81951107+00:00 | 2024-09-25 08:05:59.81951107+00:00 | | <first>.<last> | | | | |
12 | 2024-10-16 12:02:24.530535501+00:00 | 2024-10-16 12:02:24.530535501+00:00 | | <first>.<last> | | | | |
13 | 2024-11-05 05:32:40.270709176+00:00 | 2024-11-05 05:32:40.270709176+00:00 | | <first>.<last> | | | | |
14 | 2024-12-17 09:32:19.148077663+00:00 | 2024-12-17 09:32:19.148077663+00:00 | | <first>.<last> | | | | |
15 | 2024-12-20 08:11:36.352191756+00:00 | 2024-12-20 08:11:36.352191756+00:00 | | <custom> | | | | |
16 | 2025-02-10 09:43:41.061294411+00:00 | 2025-02-10 09:43:41.061294411+00:00 | | <first>.<last> | | | | |
17 | 2025-03-11 08:36:14.999339636+00:00 | 2025-03-11 08:36:14.999339636+00:00 | | | <first> <last> | <first>.<last>@<domain> | <id_2> | oidc | <url_2>
- My user seems to have been migrated correctly. I logged out and logged back in using the Tailscale client shortly after updating Headscale.
- Users 4-16 are OIDC, but haven't reauthenticated yet.
- User 17 used to be ID 3, but a new user was created upon reauthentication (after upgrading to 0.24 and then 0.25) so we destroyed the old one.
Note that the provider_identifier for me and user 17 are different. I don't know if they should be.
Just tried two things:
Reauth
Reauthenticated user 4 (not that the id matters I guess 😅) and encountered an internal server error again.
2025-03-17T12:55:36Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has disconnected, mapSession: 0xc000564480, chan: 0xc0000af8f0 node=<name>s-MacBook-Pro node.id=9 omitPeers=false readOnly=false stream=true
2025-03-17T12:55:38Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
However, shutting down Tailscale and starting it again worked, so the previous key still seems valid. The output of headscale users list is unchanged, though I haven't verified the database.
2025-03-17T12:56:21Z INF home/runner/work/headscale/headscale/hscontrol/poll.go:634 > node has connected, mapSession: 0xc000003e00, chan: 0xc0000ae9a0 node=<name>s-MacBook-Pro node.id=9 omitPeers=false readOnly=false stream=true
Registered new user
Registered a user who has never logged in to Headscale previously. They have a unique first and last name, so there should be no such collisions.
2025-03-17T12:59:15Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
2025-03-17T12:59:23Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"Bad Request\"" code=403
2025-03-17T12:59:36Z ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500```
I have the same issue. Even adding new users is creating this behavior. Does anyone has any hint on this ? Any workaround ? In this state, headscale 0.25 is just unusable
For those having the same issue, I managed to solve the user creation error :
My issue was that even with the migration flag on, my users had a user.name empty,
> headscale users list
ID | Name | Username | Email | Created
1 | | <redacted> | | 2024-04-24 03:37:43
2 | | <redacted> | | 2024-04-25 19:20:32
3 | | <redacted> | | 2024-04-28 16:01:08
5 | | <redacted> | | 2024-05-04 15:11:28
6 | | <redacted> | | 2024-05-04 16:30:55
7 | | <redacted> | | 2024-12-13 12:36:51
and most importantly, my OIDC provider was returning an empty username to headscale (I am using authelia).
The UNIQUE constraint were then failing on creating a new user with users.name equals to ''.
After digging into the code of headscale, I found that the name (in the DB) is mapped with the username sent by the OIDC provider. And after adjusting the scopes in authelia, allowing profile and username I managed to create a new user account, with a valid Name, a Username and Email.
Bottom line : Destroying and recreating user is possible, but you have to ensure that you're allowing headscale to fetch the user username to avoid duplication of empty value.
It's working for v0.24 and 0.25
I'm facing the same problem. I tried the solution mentioned in #2505: https://www.authelia.com/integration/openid-connect/openid-connect-1.0-claims/#restore-functionality-prior-to-claims-parameter, but using the claims_policy just gives me the same error. Not using the claims_policy lets me login, but without a proper username.
After the successful login the username field is empty and I get successfully authenticated as https://auth.example.org/{long-uid} in the app. Afterwards it shows this URL instead of the username and none of my other devices.
The resulting "user" in the table looks like this:
id created_at updated_at deleted_at name display_name email provider_identifier provider profile_pic_url
-- ---------------------------------- ----------------------------------- ---------- ---- ------------ ----- ------------------------------------------------------------ -------- ---------------
12 2025-04-08 17:38:38.82871823+02:00 2025-04-08 17:42:38.837385594+02:00 https://auth.{redacted}/4735db7e-eaed-4722-a3da-44b9d65df5e3 oidc
None of the other users (all use oidc) have the provider-field set.
Using Authelia 4.39.1 and Headscale 0.25.1 might be related to #2516?
We are experiencing the same error: creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067) since updating to v0.24 for creation of new users / new OIDC logins. This error is returned to the user.
Updating to v0.25 does not help. The only thing that changes is the error returned to the user is now internal server error. Headscale still logs:
ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
The only way we bypass it is to manually create the user with the cli: headscale users create .. and use v0.24, since this workaround does not work in v0.25..
This is using Google's OAuth 2.0: https://developers.google.com/identity/openid-connect/openid-connect, default scopes (openid, profile, email).
We had a similar issue in 0.24.0-beta.2 with Entra ID. The login caused a new user to be created, however, without a user name.
> headscale users list
ID | Name | Username | Email | Created
5 | John Doe | | | 2025-05-08 08:21:21
The next user who authenticated using SSO violated the UNIQUE constraint because of the empty user name. We could resolve it temporarily by adding a username to the existing user.
> headscale users rename -i 5 -r john
Same problem here with Google OIDC, I can't even rename or update user since the error cannot edit OIDC user. Does anyone know any workaround? It's blocking my user registration.
@cymonkey i ended up going into the SQLite database, and update name and updated_at in the users table to unblock user creation 🙈
Thanks @joscdk, tried and it worked, aw the only way I can think of at the this time too
@kradalby In v0.26.0 it doesn't even create the user in the DB. The server just logs:
ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
@joscdk @cymonkey what exactly did you need to do to workaround the user creation?
@ventsislav-georgiev The users.name unique constraint still exists and the OIDC authen function somehow doesn't update the name when user reauthen/register, so it actually can still create one new user via OIDC authen, the user will has a users.name = null but any other user creation attempts after that will raise the error as you've seen.
All you need to do is to find the first created user via OIDC auth and update their users.name:
$ sqlite3 /var/lib/headscale/db.sqlite
# Or if you're running in a docker: sudo sqlite3 keys/db.sqlite
sqlite> SELECT * FROM users;
# Look for the user with name empty and provider = OIDC, take note the ID
sqlite> UPDATE users SET name='<unique_username>' WHERE id='222';
Not really a workaround tbh.
Possible solution, which I tested working on our setup: https://github.com/juanfont/headscale/pull/2611
With the help of @kradalby I found that my DB indexes were not up to date with what was expected in v0.26.
I haven't tested from scratch as it is difficult for me to reset or setup a separate test instance.
My issue occurs with existing DB which was gradually updated from version 0.22.1 onward.
These were the indexes:
And it appears this migration hasn't been run or failed silently: https://github.com/juanfont/headscale/blob/d2879b2b3675c6c38b1a39dd7a7b44c1679287a9/hscontrol/db/db.go#L544
despite having it in the migrations table:
After manually fixing my indexes by dumping the DB and running the following script:
-- 1. Create a new table without the UNIQUE constraint on "name"
CREATE TABLE users_new (
id integer PRIMARY KEY,
created_at datetime,
updated_at datetime,
deleted_at datetime,
name text,
display_name text,
email text,
provider_identifier text,
provider text,
profile_pic_url text
);
-- 2. Copy data
INSERT INTO users_new
SELECT id, created_at, updated_at, deleted_at, name, display_name, email, provider_identifier, provider, profile_pic_url
FROM users;
-- 3. Drop old table
DROP TABLE users;
-- 4. Rename new table
ALTER TABLE users_new RENAME TO users;
-- 5. Re-create indices
CREATE INDEX idx_users_deleted_at ON users(deleted_at);
CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;
CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier);
CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL;
All is working properly now with 0.26 and new users logged in from the OIDC are without usernames. They are created successfully on first login.
Here are the new users without usernames as listed by the CLI headscale users list:
Thats a super helpful hint, thank you! Digging into my schema I found that it had some migrations missing, too! Unfortunately its not easy to find out which those are missing and what the sideeffects may be if I apply them now. I'm probably going to wipe my installation and create everything from scratch and see if that helps.
I am getting a similar error too, with Azure EntraID
2025-06-01T15:44:14Z ERR user msg: invalid code error="could not exchange code for token: oauth2: \"invalid_grant\" \"AADSTS54005: OAuth2 Authorization code was already redeemed, please retry with a new valid code or use an existing refresh token. Trace ID: <TRACE_ID> Correlation ID: <CORR_ID> Timestamp: 2025-06-01 15:44:14Z\"" code=403
I have not tried some of the suggested method mentioned above.
I got it working again by manually fixing the entries in the users table. Setting provider_identifier to the identifier I found in the Authelia-Logs and provider to oidc like this:
UPDATE users SET provider_identifier = 'https://auth.example.com/{some-uid}' WHERE id = 1;
UPDATE users SET provider = 'oidc' WHERE id = 1;
worked for me. Users can login again.
With the help of @kradalby I found that my DB indexes were not up to date with what was expected in v0.26.
I haven't tested from scratch as it is difficult for me to reset or setup a separate test instance.
My issue occurs with existing DB which was gradually updated from version 0.22.1 onward.
These were the indexes:
And it appears this migration hasn't been run or failed silently:
Line 544 in d2879b2
"CREATE UNIQUE INDEX IF NOT EXISTS idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL;", despite having it in the migrations table:
After manually fixing my indexes by dumping the DB and running the following script:
-- 1. Create a new table without the UNIQUE constraint on "name" CREATE TABLE users_new ( id integer PRIMARY KEY, created_at datetime, updated_at datetime, deleted_at datetime, name text, display_name text, email text, provider_identifier text, provider text, profile_pic_url text ); -- 2. Copy data INSERT INTO users_new SELECT id, created_at, updated_at, deleted_at, name, display_name, email, provider_identifier, provider, profile_pic_url FROM users; -- 3. Drop old table DROP TABLE users; -- 4. Rename new table ALTER TABLE users_new RENAME TO users; -- 5. Re-create indices CREATE INDEX idx_users_deleted_at ON users(deleted_at); CREATE UNIQUE INDEX idx_provider_identifier ON users (provider_identifier) WHERE provider_identifier IS NOT NULL; CREATE UNIQUE INDEX idx_name_provider_identifier ON users (name,provider_identifier); CREATE UNIQUE INDEX idx_name_no_provider_identifier ON users (name) WHERE provider_identifier IS NULL;All is working properly now with 0.26 and new users logged in from the OIDC are without usernames. They are created successfully on first login.
Here are the new users without usernames as listed by the CLI headscale users list:
Thank you, worked for me!
I'm working on this over in #2617 , more databases to add to the tests would be greatly appreciated. only schema helps, but with data would be even better.
I think we (@nblock) have a script for helping us randomise the data.
As this comment is a bit confusing, I posted a lengthier one here: https://github.com/juanfont/headscale/issues/2597#issuecomment-3028234284
I'm getting the same issue but with Pocket-id. ERR http internal server error error="creating or updating user: constraint failed: UNIQUE constraint failed: users.name (2067)" code=500
These are my users
1 | | lrivera | | 2023-10-18 23:34:43
2 | | rrivera | | 2024-02-08 15:48:01
I just try the @ventsislav-georgiev and it worked
I'm going to close this as fixed in the upcoming database cleanup for SQLite, no further work will be done for Postgres.