netbird Request failed with status code 401 (Authentik)

Describe the problem

After updating to authentik version 2024.10.4 I am no longer able to access the dashboard as I get an "invalid token" error. Looking at the management logs I can see the following error: management-1 | 2024-11-23T11:01:07Z WARN [context: SYSTEM] management/server/account.go:1114: failed warming up cache due to error: 403 Forbidden. I have tried deleting the Netbird service account's token and create a new one and I have also tried completely removing the application and provider and setting them up again from scratch but it didn't fix the error. With version 2024.10.2 everything worked just fine.

To Reproduce

Steps to reproduce the behavior:

Update authentik to version 2024.10.4
Check for the error in the management logs

Are you using NetBird Cloud?

Self-hosted

NetBird version

0.33.0

Screenshots

Nov 23 '24 11:11 MDMeridio001

Hello, @MDMeridio001, it seems like something went wrong with the guide steps 3 and 4 on your configuration. Can you review them and rerun the configure.sh script?

As an alternative, you can disable IdP manager in your management.json file by setting IdpManagerConfig.ManagerType and then restarting the management service with docker compose restart management

Nov 23 '24 12:11 mlsmaycon

@mlsmaycon Thanks for the reply. I'd prefer not to disable the IdP as I have all of my users configured there.

I would like to specify that it worked in version 2024.10.2 and I suspect that maybe they have made some changes to some of authentik's API endpoints. I have also checked nginx logs and it seems like error 403 is returned when the management service tries to reach this endpoint: [23/Nov/2024:12:03:53 +0100] "GET /api/v3/core/users/?page=1 HTTP/2.0" 403 58 "-" "OpenAPI-Generator/1.0.0/go". If I try to access the same page in a web broswer logged in as the Netbird service account I successfully get a list with all the users in json format.

I would also like to mention that since I followed the guide when I first set netbird and authentik up the WebUI for authentik changed significantly, so it might be in need of an update. For example, when I tried to recreate the Netbird service account the token was not created automatically and I had to manually add one.

Nov 23 '24 12:11 MDMeridio001

@mlsmaycon Just an update. I restored an old backup of authentik (version 2024.8.2) and it immediately started working again.

Nov 23 '24 12:11 MDMeridio001

The backup is old but are you running the latest authentik version?

Nov 23 '24 13:11 mlsmaycon

@mlsmaycon No, I'm running version 2024.8.2

Nov 23 '24 13:11 MDMeridio001

I am having the same problem since updating to 2024.10.4, only that rolling back to 2024.8.2 (or any other older version) does not restore functionality.

The service account mentioned in step 3 and 4 of the guide seems to work fine though, in Authentik I see it logging in successfully

I have even set up netbird from scratch, deleting all configuration and recreating it from infrastructure artifacts with Authentik verisons 2024.8.6, 2024.8.5, 2024.8.2, 2024.10.4 and 2024.10.3.

There were some issues with redirect URLs for 2024.8.5 and 2024.10.3 which since have been resolved.

Currently I am on 2024.8.6, which is the latest supported build of 2024.8. Those are the logs:

2024-11-23T21:29:44Z WARN [context: SYSTEM] management/server/account.go:1114: failed warming up cache due to error: 403 Forbidden

2024-11-23T21:30:33Z DEBG management/server/account.go:1515: account cres9lc1955s73f2aig0 not found in cache, reloading
2024-11-23T21:30:33Z ERRO [context: HTTP, requestID: 5b94c307-da2a-406f-9545-3a886a33d7c4] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: 403 Forbidden
2024-11-23T21:30:33Z ERRO [context: HTTP, requestID: 5b94c307-da2a-406f-9545-3a886a33d7c4] management/server/http/util/util.go:81: got a handler error: token invalid
2024-11-23T21:30:33Z ERRO [requestID: 5b94c307-da2a-406f-9545-3a886a33d7c4, context: HTTP] management/server/telemetry/http_api_metrics.go:168: HTTP response 5b94c307-da2a-406f-9545-3a886a33d7c4: GET /api/users status 401
2024-11-23T21:30:33Z DEBG [context: HTTP, requestID: 5b94c307-da2a-406f-9545-3a886a33d7c4] management/server/telemetry/http_api_metrics.go:181: request GET /api/users took 357 ms and finished with status 401
2024-11-23T21:30:35Z DEBG [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:120: keys refreshed, new UTC expiration time: 2024-11-23 21:30:35.465361803 +0000 UTC
2024-11-23T21:30:35Z DEBG [context: HTTP, requestID: f42c4177-9e51-4d9a-83d9-a11aacd27150] management/server/account.go:2002: overriding JWT Domain and DomainCategory claims since single account mode is enabled
2024-11-23T21:30:35Z DEBG [context: HTTP, requestID: f42c4177-9e51-4d9a-83d9-a11aacd27150] management/server/account.go:1577: looking up user 1 of account cres9lc1955s73f2aig0 in cache

The netbird service account is in the authentik-admins group:

Nov 23 '24 21:11 Spiritreader

Had the same problem. After adding api access to scopes of the Authentik OAuth2 Provider and restarting the management container seems to work again. grafik

Nov 23 '24 23:11 roehren

Had the same problem. After adding api access to scopes of the Authentik OAuth2 Provider and restarting the management container seems to work again.

You're fantastic, that worked! Thank you ❤️

Nov 24 '24 02:11 Spiritreader

Had the same problem. After adding api access to scopes of the Authentik OAuth2 Provider and restarting the management container seems to work again.

Thank you a lot, that solved the issue immediately.

Nov 24 '24 11:11 MDMeridio001

Had the same problem. After adding api access to scopes of the Authentik OAuth2 Provider and restarting the management container seems to work again.

Thank you a lot, that solved the issue immediately.

For me that wasn't the only issue I had to reconfigure the redirects under Providers. It seems authentik changed how these are entered in the latest version (they introduced strict and regex option in text fields rather than just regex and multiple lines in a text box) so in short they got wiped :-(

Nov 25 '24 20:11 rdeangel

I ran into simmilar issues. I wasnt able to login to the netbird management dashboard too with the following message:

I had to do the following steps to get it working again (as described above):

In authentik provider: 1a. change the https://netbird.tld.* to regex 1b. add 'authentik api access' to selected scopes
restart netbird management container

Netbird version: 0.33.0 Authentik version: 2024.10.4

Nov 26 '24 19:11 Nivek938

I ran into simmilar issues. I wasnt able to login to the netbird management dashboard too with the following message:

I had to do the following steps to get it working again (as described above):
1. In authentik provider:
   1a. change the https://netbird.tld.* to regex
   1b. add 'authentik api access' to selected scopes

2. restart netbird management container
Netbird version: 0.33.0 Authentik version: 2024.10.4

That worked for me. Thanks. Also, the domain redirects need to be each in their own line. Authentik's documentation displays this correctly.

Nov 27 '24 14:11 xpufx

Looks like I've hit the same thing or at least something similar. After adding both the api scope and fixing the redirect I can login but when it redirects back to netbird at /peers The call to /api/users fails with the error Error when validating JWT claims: error parsing token: invalid issuer.

I went several rounds trying to figure this out but the JWT looks fine, issuer is my authentik app URL.

Nov 28 '24 07:11 btilford

Looks like I've hit the same thing or at least something similar. After adding both the api scope and fixing the redirect I can login but when it redirects back to netbird at /peers The call to /api/users fails with the error Error when validating JWT claims: error parsing token: invalid issuer.

I went several rounds trying to figure this out but the JWT looks fine, issuer is my authentik app URL.

@btilford Did you figure this out? I'm running into the same thing today.

Dec 02 '24 03:12 CD11b

@CD11b not yet. I'm thinking maybe the instructions on setting up the Oauth in Authentik should be using client credential grants like normal instead of going and creating service accounts. Haven't had a chance to test yet though.

Dec 02 '24 21:12 btilford

Hi, In addition to what's listed above, I had to do the following:

Replace an expired certificate-key pair: From System > Certificates, generate a new pair and update my provider to use it as the Signing key.
Replace a service account with an expired token: Create a new service account and update management.json (.IdpManagerConfig.ExtraConfig) accordingly.
Set .HttpConfig.IdpSignKeyRefreshEnabled to true in management.json.

Probably the issue I met is different from OP's and likely stems from my year-old setup, though.

netbirdio/management:0.33.0
netbirdio/dashboard:v2.7.1
ghcr.io/goauthentik/server:2024.10.4

Dec 04 '24 10:12 likeablob

Should the traefik docker compose have a relay server like the non traefik compose?

Dec 12 '24 02:12 btilford

this fixed it for me (1st part):

https://github.com/netbirdio/netbird/issues/1657#issuecomment-2127732511

Dec 16 '24 21:12 ne0YT

The same problem Redirect URI error

change in my authentik:

and

Now the access dashboard is work :)

Dec 20 '24 10:12 barto95100

Regarding 401 - Invalid Token: I have read through many threads, verified all settings, tested alternate configurations based on the threads and cannot get rid of this error. I see that a few still have the error and wondering if any progress has been made?

I'm getting permissions to run in Authintik, but when routed back to the netbird dashboard (oobe screen, never ran successfully) I get stuck at this:

It seems that most/all of the services requesting a token are not getting one. ..failed warming up cache, and JWT errors, etc.
Authentik ver: 2024.12.1 Ports: 9000->9443 Netbird ver: latest 33073->443

These are both docker containers on the same server with different sub domain routes. auth..cloud and nb..cloud

I'm not accustomed enough yet to get the logs I would really want to see yet. I would also like to run a sniffer to see the outgoing and incoming packets, but yet again.... not educated enough on linux. Hmmmm... maybe I should build a Windows VM and recreate the configuration to see if I get the same errors. I don't believe it's a Linux thing, just more proficient in Windows internals.

Any more suggestions would be greatly appreciated.

management-1 | 2024-12-25T02:10:10Z ERRO [context: HTTP, requestID: 0df096b5-7006-4ad5-9791-f0cded5f59d3] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-12-25T02:10:10Z ERRO [context: HTTP, requestID: 0df096b5-7006-4ad5-9791-f0cded5f59d3] management/server/telemetry/http_api_metrics.go:168: HTTP response 0df096b5-7006-4ad5-9791-f0cded5f59d3: GET /api/users status 401 management-1 | 2024-12-25T02:10:11Z ERRO [context: HTTP, requestID: d9beb883-8d54-4a28-b378-6ea4d6981660] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400 management-1 | 2024-12-25T02:10:11Z ERRO [context: HTTP, requestID: d9beb883-8d54-4a28-b378-6ea4d6981660] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-12-25T02:10:11Z ERRO [context: HTTP, requestID: d9beb883-8d54-4a28-b378-6ea4d6981660] management/server/telemetry/http_api_metrics.go:168: HTTP response d9beb883-8d54-4a28-b378-6ea4d6981660: GET /api/users status 401 management-1 | 2024-12-25T02:10:17Z ERRO [context: HTTP, requestID: 23e15c63-ee40-4022-ba81-9f9669504577] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400 management-1 | 2024-12-25T02:10:17Z ERRO [context: HTTP, requestID: 23e15c63-ee40-4022-ba81-9f9669504577] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-12-25T02:10:17Z ERRO [context: HTTP, requestID: 23e15c63-ee40-4022-ba81-9f9669504577] management/server/telemetry/http_api_metrics.go:168: HTTP response 23e15c63-ee40-4022-ba81-9f9669504577: GET /api/users status 401 management-1 | 2024-12-25T02:10:31Z ERRO [context: HTTP, requestID: dd20e868-1565-47ba-b9cb-f7ec1a8d6285] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400 management-1 | 2024-12-25T02:10:31Z ERRO [requestID: dd20e868-1565-47ba-b9cb-f7ec1a8d6285, context: HTTP] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-12-25T02:10:31Z ERRO [requestID: dd20e868-1565-47ba-b9cb-f7ec1a8d6285, context: HTTP] management/server/telemetry/http_api_metrics.go:168: HTTP response dd20e868-1565-47ba-b9cb-f7ec1a8d6285: GET /api/users status 401 management-1 | 2024-12-25T02:10:35Z ERRO [context: HTTP, requestID: 981f746e-d787-44c4-8a38-24eff7d1eb38] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400 management-1 | 2024-12-25T02:10:35Z ERRO [context: HTTP, requestID: 981f746e-d787-44c4-8a38-24eff7d1eb38] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-12-25T02:10:35Z ERRO [context: HTTP, requestID: 981f746e-d787-44c4-8a38-24eff7d1eb38] management/server/telemetry/http_api_metrics.go:168: HTTP response 981f746e-d787-44c4-8a38-24eff7d1eb38: GET /api/users status 401 management-1 | 2024-12-25T02:11:03Z ERRO [context: HTTP, requestID: 4fa80205-ba1c-402a-9ddf-e6d65f4ebb7a] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: unable to get authentik token, statusCode 400 management-1 | 2024-12-25T02:11:03Z ERRO [context: HTTP, requestID: 4fa80205-ba1c-402a-9ddf-e6d65f4ebb7a] management/server/http/util/util.go:81: got a handler error: token invalid management-1 | 2024-12-25T02:11:03Z ERRO [requestID: 4fa80205-ba1c-402a-9ddf-e6d65f4ebb7a, context: HTTP] management/server/telemetry/http_api_metrics.go:168: HTTP response 4fa80205-ba1c-402a-9ddf-e6d65f4ebb7a: GET /api/users status 401

Dec 25 '24 02:12 seamajr

Reading through the Authentik docs, I noticed that Authentik 2024.10.x introduced encrypted JWT, which are on by default. I turned them off in the Netbird provider and now it works. grafik

Dec 25 '24 23:12 FabulousCodingFox

The same problem Redirect URI error

change in my authentik:

and

Now the access dashboard is work :)

Reading through the Authentik docs, I noticed that Authentik 2024.10.x introduced encrypted JWT, which are on by default. I turned them off in the Netbird provider and now it works.

After I applied these changes, it worked. Thank you very much. This should be added to troubleshooting of Netbird or Authentik.

Jan 11 '25 12:01 bannert1337

Reading through the Authentik docs, I noticed that Authentik 2024.10.x introduced encrypted JWT, which are on by default. I turned them off in the Netbird provider and now it works.

How did you disabled encrypted JWT, I select --------, and save the self-signed cert is still there, it seems I'm unable to deselect it.

or did you disable it some other way?

Jan 26 '25 11:01 rdeangel

Reading through the Authentik docs, I noticed that Authentik 2024.10.x introduced encrypted JWT, which are on by default. I turned them off in the Netbird provider and now it works.

How did you disabled encrypted JWT, I select --------, and save the self-signed cert is still there, it seems I'm unable to deselect it.

or did you disable it some other way?

You are trying to disable the signing key. This needs to stay. You need to disable the encryption key below it.

Jan 26 '25 11:01 FabulousCodingFox

based on this:

my encryption key is not select and signing should be the same as before (they do need to be signed).

having like this which is what I had to start with should disable JWT encryption right?

Yea it works, I had messed up one of my env variables so I also broke it cause of that turns out. All working now.

Jan 26 '25 11:01 rdeangel

The same problem Redirect URI error

change in my authentik:

and

Now the access dashboard is work :)

This worked for me - thanks :)

Jan 31 '25 23:01 bsmithuk

Since I ended up here after encountering the same problem, and struggled to make it work even with the information of this thread, I will add this. If you created a group in Authentik to limit the access to netbird to a subset of users, do not forget to add the Netbird service account to this group. It is what solved the problem for me. :)

Feb 14 '25 22:02 Berjou

same problem again 401 token invalid... :(

management-1  | 2025-02-17T17:45:47Z ERRO [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:142: getPublicKey error: unable to find appropriate key
management-1  | 2025-02-17T17:45:47Z ERRO [context: HTTP, requestID: 8f31c841-aeb8-4923-9c93-a17a475fe29a] management/server/jwtclaims/jwtValidator.go:180: error parsing token: unable to find appropriate key
management-1  | 2025-02-17T17:45:47Z ERRO [context: HTTP, requestID: 8f31c841-aeb8-4923-9c93-a17a475fe29a] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: error parsing token: unable to find appropriate key
management-1  | 2025-02-17T17:45:47Z ERRO [context: HTTP, requestID: 8f31c841-aeb8-4923-9c93-a17a475fe29a] management/server/http/util/util.go:85: got a handler error: token invalid
management-1  | 2025-02-17T17:45:47Z ERRO [context: HTTP, requestID: 8f31c841-aeb8-4923-9c93-a17a475fe29a] management/server/telemetry/http_api_metrics.go:189: HTTP response 8f31c841-aeb8-4923-9c93-a17a475fe29a: GET /api/users status 401
management-1  | 2025-02-17T17:45:50Z ERRO [context: SYSTEM] management/server/jwtclaims/jwtValidator.go:142: getPublicKey error: unable to find appropriate key
management-1  | 2025-02-17T17:45:50Z ERRO [requestID: f0beb3c0-0a37-47f2-af34-c1e3cef6d744, context: HTTP] management/server/jwtclaims/jwtValidator.go:180: error parsing token: unable to find appropriate key
management-1  | 2025-02-17T17:45:50Z ERRO [requestID: f0beb3c0-0a37-47f2-af34-c1e3cef6d744, context: HTTP] management/server/http/middleware/auth_middleware.go:89: Error when validating JWT claims: error parsing token: unable to find appropriate key
management-1  | 2025-02-17T17:45:50Z ERRO [context: HTTP, requestID: f0beb3c0-0a37-47f2-af34-c1e3cef6d744] management/server/http/util/util.go:85: got a handler error: token invalid
management-1  | 2025-02-17T17:45:50Z ERRO [context: HTTP, requestID: f0beb3c0-0a37-47f2-af34-c1e3cef6d744] management/server/telemetry/http_api_metrics.go:189: HTTP response f0beb3c0-0a37-47f2-af34-c1e3cef6d744: GET /api/users status 401

I don't understand :(

Feb 17 '25 17:02 barto95100