Refreshing tokens for authorization code flow does not work with dynamic hostname and different front- and backchannel url

[NilsEngelbach]

Before reporting an issue

• [X] I have read and understood the above terms for submitting issues, and I understand that my issue may be closed without action if I do not follow them.

Area

core

Describe the bug

Refresh tokens can not be refreshed when using a different front- and backchannel url in the authorization code flow, because the refresh token request throws an error (invalid issuer).

Detailed Description: We are using keycloak without a fixed hostname and without hostname-strict. This means the issuer is resolved dynamically based on the request headers.

The applications are using the authorization code flow with different urls for the frontend redirect (public) and the backchannel communication (internal) to obtain the token / refresh the token.

The tokens and the session are based on the public frontend url when redirecting to the authorization endoint. This means the issuer in the token is based on the request (e.g. http://192.168.XXX.XXX/auth/realms/<realm>, when accessed via local IP).

Since we are using the internal backchannel url for the token refresh at the token endpoint, keycloak throws an error that the issuer of refresh token (http://192.168.XXX.XXX/auth/realms/<realm>) does not match the expected issuer (http://keycloak:8080/auth/realms/<realm>).

I think the check was introduced with this Pull Request: https://github.com/keycloak/keycloak/pull/24212

Maybe it would be a solution to only check the issuer when the hostname-strict option is enabled?

Why can't we use a fixed hostname?

We can not / do not want to set a fixed `hostname`, since the system should be accessible within different networks as shown in the diagram (e.g. changing IP Adresses within a VPN):

Version

23.0.6

Regression

• [ ] The issue is a regression

Expected behavior

Allow using different front- and backchannel urls for the authorization code flow.

Actual behavior

Refresh token requests throws error that the issuer is not expected.

How to Reproduce?

Start keycloak without a fixed hostname and without hostname-strict. This means the issuer is resolved dynamically based on the request headers.

Create a client and use different front- and backchannel urls for the authorization code flow: Authorization URL: (e.g. relative url for browser redirect) /auth/realms/<realm>/protocol/openid-connect/auth Token URL: (e.g. internal docker network to obtain the tokens) http://keycloak:8080/auth/realms/<realm>/protocol/openid-connect/token

Anything else?

Related Issues: https://github.com/keycloak/keycloak/issues/26017 https://github.com/keycloak/keycloak/issues/22191

Related Pull Request: https://github.com/keycloak/keycloak/pull/24212

[NilsEngelbach]

Just tested with newest release 24.0.1 and it can still be reproduced:

2024-03-07 13:20:47,050 INFO  [io.quarkus] (main) Keycloak 24.0.1 on JVM (powered by Quarkus 3.8.1) started in 32.026s. Listening on: http://0.0.0.0:8080
2024-03-07 13:20:47,051 INFO  [io.quarkus] (main) Profile prod activated.
2024-03-07 13:20:47,051 INFO  [io.quarkus] (main) Installed features: [agroal, cdi, hibernate-orm, jdbc-postgresql, keycloak, logging-gelf, micrometer, narayana-jta, reactive-routes, resteasy-reactive, resteasy-reactive-jackson, smallrye-context-propagation, smallrye-health, vertx]
2024-03-07 13:22:48,285 WARN  [org.keycloak.events] (executor-thread-2) type="REFRESH_TOKEN_ERROR", realmId="XXX", clientId="XXX", userId="null", ipAddress="172.18.XXX.XXX", error="invalid_token", grant_type="refresh_token", client_auth_method="client-secret"

[mposolda]

@vmuzikar Is it ok to re-assign this to cloud-native team as I suppose this is something, which might be improved with the hostname work cloud-native team has in the backlog? Feel free to re-assign back to core-clients if this is unrelated.

[NilsEngelbach]

Would it be sufficient to only check the issuer in the token refresh when hostname-strict is set? I can also try to contritbute the change, if you think the solution is feasible.

[MarinaTeper35]

Hello, we're facing the same issue ,is there any estimation for fix/ enhancement of the issue? Thanks!

[Jayashree-Rajendran]

We are facing same issue, any update on this??

[shawkins]

@mposolda this needs core-clients involvement, the changes for #22191 are what introduced this check and the new hostname changes just mean that we'd be consulting hostname-strict-backchannel here instead.

We likely need to follow the suggestiong in https://github.com/keycloak/keycloak/issues/27660#issuecomment-2006047100 to disable the iss verification in a non-strict scenario, or have the user configure what are all the possible backchannel hostnames to verify against.

cc @vmuzikar

[shawkins]

~priority-normal

[keycloak-github-bot[bot]]

Due to the amount of issues reported by the community we are not able to prioritise resolving this issue at the moment.

If you are affected by this issue, upvote it by adding a :thumbsup: to the description. We would also welcome a contribution to fix the issue.

[vmuzikar]

Using dynamically resolved front- and backchannel (while the two being different URLs) is unsupported with the new Hostname v2 provider which will be the default in Keycloak 25. Please see the new docs. Keycloak needs to know what is the frontend URL when a Client access the OIDC discovery endpoint, otherwise it would not even know the URL of authentication endpoint to redirect the browser, among other issues like the one with issuer validation. It was basically a bug that it worked with the Hostname v1 implementation. I believe we should reject this. WDYT @shawkins?

[shawkins]

@vmuzikar my mistake, I had not fully understood that with the new provider that the issuer and expected issuer would both resolve to the fixed front-end url. In looking at the V1 code I had thought that the expected issuer could still resolve to the dynamic url. It does seem like this can be closed then.

[shawkins]

May have been too quick to close. Since the user explicitly does not want a fixed front-end, this case is still not supported with the new hostname implementation.

This also overlaps with #17634

[vmuzikar]

May have been too quick to close. Since the user explicitly does not want a fixed front-end, this case is still not supported with the new hostname implementation.

Not sure what you mean? :) We cannot support multiple front-end URLs.

[shawkins]

Not sure what you mean? :) We cannot support multiple front-end URLs.

If you want to resolve this as a duplicate of #17634 I fine with that - if #17634 remains open then we should ensure this case is addressed by that as well.

[vmuzikar]

@shawkins IMHO #17634 is not possible to do. As I mentioned, having non-static frontend url (or having multiple frontend urls) when accessing Keycloak via backend is not something we can do as Keycloak could not determine which frontend url to use when accessed via backend.

[shawkins]

@vmuzikar @mposolda to understand a little more if hostname-strict=false, then the expectation is that any token you obtain has to be used via the same url so the issuer matches?

Is a possible workaround / solution for this issue and the docker example from #17634 to make the internal domain used for keycloak, in this example keycloak, the hostname and make it resolvable by clients in each network so that it's effectively public?

Otherwise if you cannot make the hostname public or effectively public, then we can't support this scenario?

[vmuzikar]

if hostname-strict=false, then the expectation is that any token you obtain has to be used via the same url so the issuer matches?

hostname-strict=false effectively enables dynamic hostname resolution. So yes, I'd say that it is expected that a client is accessing Keycloak via the same URL. It does not mean some other client cannot use a different URL but it'll have a different iss from the first client.

Is a possible workaround / solution for this issue and the docker example from #17634 to make the internal domain used for keycloak, in this example keycloak, the hostname and make it resolvable by clients in each network so that it's effectively public?

Yes, I'd say front-end URL needs to be public.

@mposolda Does it make sense from the OIDC perspective?

[rmartinc]

@vmuzikar I'm trying to understand what you are saying:

hostname-strict=false effectively enables dynamic hostname resolution. So yes, I'd say that it is expected that a client is accessing Keycloak via the same URL. It does not mean some other client cannot use a different URL but it'll have a different iss from the first client.

Now this is --hostname-backchannel-dynamic=true, right? Doing some tests enabling this option, the backend URL is allowed to be dynamic and it allows to connect from internal networks. The iss remains the same so the OIDC clients work as expected (I mean, the iss announced in the well-known address continue being the frontend URL and it's teh same to teh one assigned to users' tokens).

Yes, I'd say front-end URL needs to be public.

Yes, exactly, the frontend url should be public and accessibly for the browser login. The OIDC client will connect internally using the backend URL which is dynamic. The iss will be the public URL but the same in the well-known (so the client is not complaining).

Just one comment here. I have tried to start the server like this with main (or a recent main, probably from yesterday or two days ago). And I cannot configure different admin URL.

./kc.sh start-dev --hostname=http://localhost:8080 --hostname-admin=http://127.0.0.1:8080 --hostname-debug=true --hostname-backchannel-dynamic=true

I see the initial login is sent to 127.0.0.1 but then uses localhost. Have you tried this? I think this is not working and I'm not sure what we should expect here.

[vmuzikar]

@rmartinc Thank you for your input!

Now this is --hostname-backchannel-dynamic=true, right?

Almost right. :) --hostname-strict is still there with v2. Basically the difference is --hostname-strict=false enables dynamic resolution for all URL types but it assumes the same URL is used everywhere, in other words that the server is accessed via the same URL at least for a given user (to have a correct iss everywhere e.g.). --hostname-backchannel-dynamic enabled dynamic resolution just for the backend and enforces it is hardcoded for frontend.

I see the initial login is sent to 127.0.0.1 but then uses localhost.

I think you've found a bug. :) Created https://github.com/keycloak/keycloak/issues/29641.

Yes, exactly, the frontend url should be public and accessibly for the browser login.

So it sounds we need to reject this issue, right?

[rmartinc]

Almost right. :) --hostname-strict is still there with v2. Basically the difference is --hostname-strict=false enables dynamic resolution for all URL types but it assumes the same URL is used everywhere, in other words that the server is accessed via the same URL at least for a given user (to have a correct iss everywhere e.g.). --hostname-backchannel-dynamic enabled dynamic resolution just for the backend and enforces it is hardcoded for frontend.

Ah! Ok! So this option is used with no hostname like in the start-dev. It uses the incoming request to know the server and it cannot be used in combination with hostname-backchannel. Understood.

So it sounds we need to reject this issue, right?

With the hostname V2 there are more options now. The hostname-backchannel-dynamic allows clients to use the private network, the only requirement is the final users has common login URL that is returned in the tokens and well-known address endpoint. I think this option would be enough for this specific issue. So yes, we can reject the issue. Let's see if there are more complaints.

[vmuzikar]

With the hostname V2 there are more options now.

I'd say that with v2 there're actually less options but they should be more clear, especially in terms of use cases we support. And having both front- and backend dynamic is unfortunately not one of them.

@rmartinc Thanks for the feedback, closing this issue.

[NilsEngelbach]

The new settings really simplify the configuration, but I am still facing the same issues.

I want the "Frontend URLs" to be resolved based on the forwarded proxy headers and I want to enable "Backend" access via the internal network (e.g. for token refresh) at the same time. But KC_HOSTNAME_BACKCHANNEL_DYNAMIC can not be used without a dedicated Hostname, which I don't have because it is resolved from the forwarded proxy headers.

This is the configuration I would like to use:

"KC_PROXY_HEADERS": "xforwarded",  # Resolve hostname dynamically for frontends
"KC_HOSTNAME_BACKCHANNEL_DYNAMIC": "true",   # Resolve hostname dynamically for backend access
"KC_HOSTNAME_STRICT": "false",  # Disable hostname checks (e.g. when refreshing tokens)
"KC_HTTP_RELATIVE_PATH": "/idp",

For me this "Bug" was introduced with this pull request: https://github.com/keycloak/keycloak/pull/24212

The refresh token endpoint should not validate the hostname against the issuer when KC_HOSTNAME_STRICT is set to false.

[vmuzikar]

@NilsEngelbach If frontend URL (specified by the --hostname option) is not specified and is dynamically resolved, backend is dynamically resolved as well. Keep in mind that when an app accesses Keycloak via backend in this case, Keycloak does not know what is the actual real frontend URL and assumes it's the same as backend (in other words, frontend will be resolved to internal URL).

[NilsEngelbach]

Let me ask another way around.

Why does the refresh token endpoint validate if the hostname it was called at, matches the issuer of the provided refresh token? The tokens are signed with the private keys of the keycloak, that means they can not be forged.

[vmuzikar]

That would be probably a question for @rmartinc. I believe the reasoning is described in https://github.com/keycloak/keycloak/issues/22191.

[NilsEngelbach]

Can we please reopen the issue, since the token refresh is still broken? It is also unintuitive that the initial token request does not do this validation. This means your application works fine until there is the first token refresh.

The Problem described in https://github.com/keycloak/keycloak/issues/22191 is related to changing the hostname of the idp (which is not quite a common usecase i think). Maybe it would have been an option to generate new keys when chaning the hostname, so that all tokens with the old hostname get invalidated.

[vmuzikar]

@NilsEngelbach I understand you concerns. That said, I'm not sure I'd consider this a bug or broken behaviour. It's simply unsupported use case due to security reasons. @rmartinc WDYT?

[NilsEngelbach]

Sorry to bug you about this topic so much 🙈 I just want to use the latest keycloak version and this issue prevents me from using it. The issue #22191 is not about security . The hostname check at the refresh token endpoint does not provide additional security, because the signature check should be enough.

[vmuzikar]

The hostname check at the refresh token endpoint does not provide additional security, because the signature check should be enough.

@mposolda @rmartinc Do you have any input on this?

[rmartinc]

For me this is the never ending story I have to say. This check for the issuer is more or less established by spec. See Issuer Identifier:

OpenID Connect supports multiple Issuers per Host and Port combination. The issuer returned by discovery MUST exactly match the value of iss in the ID Token.

And in oidc discovery spec it says:

The issuer value returned MUST be identical to the Issuer URL that was used as the prefix to /.well-known/openid-configuration to retrieve the configuration information. This MUST also be identical to the iss Claim value in ID Tokens issued from this Issuer.

So this check should be there and it should be done with default keycloak configuration (which is hostname strict, spec compliant by default). The problem is that people sometimes do not configure the same name for internal and external access which, IMHO, it's the normal way of working with OIDC. So they want an exception to allow different names for internal/external access. And that's the reason to have all this messy configuration of --hostname-strict and --hostname-backchannel-dynamic and all the exceptions. Obviously google, facebook and other SaaS providers never have this problem as they have always the same name for everybody.

What @NilsEngelbach is complaining is that previously refresh token was not checking this and now it's doing it. But this check was already in other places like the introspect endpoint for example (here). And that's the root cause for issue #22191. So if a internal client needs to use introspect it would have the same issue.

The idea is that keycloak offers some configuration to relax the issuer checking. AFAIK (because this is changing all the time) currently keycloak has the following options:

1. hostname-strict, default way and the name should always be the same for everybody.
2. hostname-backchannel-dynamic, this option allows two different names (against the spec) for internal/external access but it should be always the same.
3. hostname-strict disabled, this is the same used in dev mode, and the issuer is just obtained from the connection. The limitation with this one is that the checks for names complicate the situation and it enforces the name should be the same for internal/external access pairs.

Obviously we can change the behavior for (2) and (3) because we are relaxing the checking. This is what we are doing all the time, with all the options. But for the moment this is how keycloak works. If we want something different we should define it again and try to make people understand the situation.

[amit-shrestha]

@NilsEngelbach Did you manage to use the latest version and get the token refresh to work? I have similar issues and cannot use the same hostname.