Admin password gets outdated when Keycloak is recreated having external database turned on

[christianviana]

Describe the bug

When using an external database option for the Keycloak resource, we are observing the following issue:

When the Keycloak resource is deleted and afterwards recreated, the admin user's password in the database is not updated with the respective value in the secret called "credentials-<CR-Name>".

Version

1.16.0

Expected behavior

The admin password is expected to be updated, so we can use it at the keycloak web console and through REST API calls.

Actual behavior

The admin password is not getting updated, and we cannot login into Keycloak's web console nor through REST API calls.

How to Reproduce?

• Configure an deploy a Keycloak resource using the external database option
• At this point it's possible to login to the Keycloak web console using the password stored in the secret named "credentials-<CR-Name>".
• Delete the Keycloak resource
• At this point the stateful set and respective pods should get destroyed, but the external database remains untouched
• Recreate the Keycloak resource
• At this point, "credentials-<CR-Name>" secret will receive new values, but the corresponding data (admin password) is not updated in the Kecloak's database. The result is we cannot login to the web console usign these crentials and CR are not getting created because they try to login using this non-working admin password stored in the secret.

Anything else?

• We have found a workarond for this: everytime the problem occurs, we can get rid of it deleting the admin user information in the database using these queries:

delete from credential
where user_id = (select id from user_entity where username = 'admin');

delete from user_role_mapping
where user_id = (select id from user_entity where username = 'admin');

delete from user_entity where username = 'admin';

Then, we have to delete and recreate the Keycloak resource, so the admin password is reinserted in the database with the new values

• Related issue at Keycloak's issue board: https://issues.redhat.com/browse/KEYCLOAK-19841

[venkyhodigere]

Is there a preference on how we want to deal with this bug?

Some Options: a) If username is admin, recreate this user with the new password in the external DB. b) Have a parameter in the keycloak CRD that says something like recreateAdminUser: bool to control this behavior. c) Always recreate users (all), irrespective of admin or not, if a request is made to create them.

Option A would be a breaking change from behavior perspective. And will give special meaning to "admin" username. But will offer the behavior one would expect like @christianviana explained in the bug report.

Option B will allow for a non-breaking change keeping existing behavior (of admin password not getting updated). However will add complexity to the code and the spec.

Option C is again a breaking change in terms of behavior. Will be consistent across all users and not just admin.

All these options will probably rely on modifying this if condition appropriately. https://github.com/keycloak/keycloak/blob/4136bf77003b2c6ff3952005f78c2032685cabf5/services/src/main/java/org/keycloak/services/resources/KeycloakApplication.java#L347

[christianviana]

Hi Venky, thank you for asking!

I don't know for other scenarios, but from my perspective, I think a great solution would be: testing if the admin user is already there, and if so, only to update his password. I don't think we need to always recreate the admin user, only if he is not found. Always recreate him would have the bad side-effect of loosing all the changes that may have been made to him.

I think the special treatment for the admin user is justified by the fact that the operator already treats him differently from other users, as his credentials are stored in the secret. And that's why we need the database credentials to be in sync with the ones stored in the secret admin.

So, I propose this "option D", that would be having a parameter called something like updateAdminPassword: bool. If parameter=yes, always update the password in db to the value in secret, meaning that the secret would be considered as the single point of truth for the admin password. The operator already checks if admin user exists and creates it if not. That behaviour wouldn't need to be changed.

[manu11th]

@venkyhodigere @christianviana I would also prefer the solution of @christianviana , just updating the password with the current value from the secret.

But I would not introduce a new parameter to control that behaviour, because the admin user credentials should always be in sync or is there a scenario, where you want to have the old credentials in the db and the new in the secret?

As fas as I understand the operator also uses the admin user to make requests to keycloak. And if the credentials in the secret don't match the credentials in the DB, the operator can't "operate" the keycloak anymore, which is a serious problem because without manual DB access, the operator would be useless in this scenario.

[vmuzikar]

Thank you for reporting this issue. Yes, I can confirm it is a known bug, actually reported a while back as KEYCLOAK-19841.

As already correctly identified in this thread, there is no simple fix as the operator needs to have correct admin credentials to reconcile Keycloak resources. A workaround is to manually (re)create the credentials-{cr-name} Secret before the recreation of Keycloak CR.

Unfortunately, due to relatively low impact (workaround exists) and high complexity of a fix (and given the new operator is just around the corner), we are not able to prioritize this issue at the moment, and would welcome community contributions. If you feel this issue has a higher priority please open a discussion on the Keycloak Developer Mailing List.

Thank you for your understanding.

[dputnamfr]

@vmuzikar How about a method to inform the operator 'don't overwrite the credentials-{cr-name} secret if it already exists?'

I'm in the same boat as these folks, but I'm using tooling to create the secret and keep it synced to our secrets store. Then any time this dang keycloak operator shows up, it takes control and overwrites the secret.

That allows the users to decide how to pass an admin user/password to keycloak, instead of trusting keycloak to generate (and then later lose) the admin password.

[MarkCupitt]

We have run into this problem as well, we use Hashicorp as our keystore and prefer to 'give" keycloak the password to use via a secret, then when it changes on rotation, (someone leaves, etc) the database will get updated automatically.

Having to script the suggested cred deletion in the database manually and then re-deploying keycloak, is way from ideal as it means we need to shutdown our whole application, which uses keycloak extensively

Its almost a deal breaker for us, but we love keycloak and if anyone has better suggestions, we would love to hear them

I'm wondering if we there was a way to dig the password out of the secret after a Hashicorp update via the external secrest operator, maybe on a post update hook, then add the password into the db, not sure if this is viable as I presume the password is encrypted, which adds to the complexity of dong it externally and may make this approach impossible

Any suggestions, ideas VERY welcome

[stianst]

Thanks (again) for reporting this issue. Keycloak 19 was the last version that included this legacy Operator, and with the release of Keycloak 20 the Operator reached EOL and this repository will be archived, please see our blog post on this topic. If this issue is still valid for the Realm Operator, please re-open it there. Thanks for your understanding. And be sure to check out our new Operator!