bitwarden
SSO login returns token missing in keyring error
Steps To Reproduce
- Go to 'bitwarden.yourdomain.com
- enter login info for you org
- Select sign in with SSO
- Enter Org ID
-
- Error returns
Expected Result
SSO logs in user and requests master password
Actual Result
Error There was an unexpected error during single sign-on.
The key {4991ac6e-606d-451d-ae14-72ddb5ddbc48} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning
Screenshots or Videos
No response
Additional Context
Works for most users, specific to a handful with no discernable similarities.
Users that fail SSO are still able to login with regular username/password.
Githash Version
{"version":"2023.4.3","gitHash":"8d9ca424-dirty","server":null,"environment":
Environment Details
Kubernetes deployment with BW_ENABLE_SSO set to True
Database Image
No response
Issue-Link
https://github.com/bitwarden/server/issues/2480
Issue Tracking Info
- [X] I understand that work is tracked outside of Github. A PR will be linked to this issue should one be opened to address it, but Bitwarden doesn't use fields like "assigned", "milestone", or "project" to track progress.
Hey there,
There may be a small possibility of a misconfiguration of your SSO configuration as I am unable to reproduce this issue, but it has been escalated for further investigation. If you have more information that can help us, please add it below.
As we use GitHub issues as a place to track bugs and other development related issues. I would also suggest reaching out to our support for additional troubleshooting here to confirm any misconfiguration in settings aren't present.
Thanks!
We were finally able to pinpoint how to recreate this issue:
- Login with SSO < This works as expected
- Close your tab without logging out
- Reattempt SSO login
- Error received stating "Token not found in keyring"
It seems the user's session token is persisting when the tab is closed instead of terminating it on the bitwarden side and allowing a new session to be authenticated.
There could be a setting we're missing? It appears the logout url field isnt yet supported.
Final Update:
It seems we solved the problem. We're currently utilizing two geographically distinct instances of Bitwarden Unified in Kubernetes with a SQL backend.
In order to achieve a highly available setup, we had to manually replicate the /etc/bitwarden/data-protection directory to the persistent volumes at both locations. This allowed users who had accounts on one instance to login to the other instance without recreating their account.
Questions we have for the BW team really are: What do the keys stored in the xml directory accomplish? Are we compromising security through copying these? Is there a better way? Do those keys rotate at any point or expire? Is there a way to persist these into the database itself to ensure access is maintained?
Im facing the same problem. User not able to login into Bitwarden via SSO.
Error :
System.Security.Cryptography.CryptographicException: The key {31b652d8-2824-4097-8b16-97c3a592e1b3} was not found in the key ring. For more information go to http://aka.ms/dataprotectionwarning
Not sure what steps were taken previously, however, if you lost the keys in the directory I mentioned above, I believe we had to wipe our database to fix it. Those keys seem to be very important for users.
Would like for the bitwarden team to explain what these keys are for to help understand the best way to securely store them.
@djchateau or any BW dev/engineer types here: just wondering if you had any insight on my last comment? We have been able to replicate the keys into the directory but it would be really valuable to understand what the keys are used for specifically and what triggers the keys to be created. We've noticed that keys have been created in August and November randomly. We originally deployed this in May.
I am also having the same issue outside an SSO context, kubernetes server has the key needed, I can't run this app locally without including this data protection key
Hello! I wanted to first clarify if there were still any questions about what the DataProtection key is used for, and also to clarify what use cases are experiencing this issue with the Unified deployment.
First, the DataProtection key is used by the application for various server-side encryption scenarios. For SSO in particular, we encrypt the SSO identifier and organization ID on the server and pass that encrypted payload through the SSO redirect process, to ensure that it is not accessible to any third parties. It is used for other things as well, as @jphillips05 noted.
Second, in response to this comment, the HA scenario that is described here is not currently a supported use-case for Unified, and it may explain the issue you're seeing. Have any users experienced this issue without the manual replication of the /etc/bitwarden/data-protection directory? We want to make sure we correctly direct our investigation.
Hey @trmartin4 thanks for some of the clarity. Happy to add some context around the ask and where we're at today.
Tracking fully on 1.
For 2: First bit of clarification for the requirement: our team has a unique requirement to provide on-premise, cross-region highly available technologies meaning if US East were to go offline, services need to instantaneously redirect without human interaction to US West. For arguments sake lets call US East primary, and US West our secondary for the rest of this post.
On to our current state:
We've since switched to the Self-Host helm chart as our deployment method since it's now a supported option and much more native to K8s. We continue to solve the aspnet key issue through a scheduled pipeline (nightly) (gitlab ci as our code repo). You can see the code snippet below for the very basic kubectl exec and cp utilizing artifacts between jobs to both region's backing PVs for the BW pods. Just to be fully clear here - the database layer is also replicate so essentially a mirror database with two instances of the BW application pointing to it. This was the question around how/when/where exactly the keys were used.
We consistently see users with the issue mentioned if we do not replicate the keys through repetitive testing. They are only able to access after we replicate the keys across to the opposite region's PVs. You also have to restart the pods after updating the underlying FS for the app to recognize the new keys. While not officially supported, we don't have other options to solve our current requirement for operations.
Open to any suggestions or thoughts from the dev team if code changes are possible to solve this in a more supportable manner! It would still be helpful to understand the triggers to create new keys so we can ensure we don't miss replicating one. Finally, is the BW platform (SAAS offering) currently available in a cross-region HA or DR capability? If so, how is that being accomplished?
CI File with stages to replicate keys utilizing gitlab artifacts. ie a poor man's rsync:
stages:
- apps-aspnet-key-match
- apps-deploy-kubectl
- apps-deploy-helm
apps-backup-region2-aspnet-keys:
stage: apps-aspnet-key-match
image: $K8S_DEPLOY_IMAGE
tags:
- region2-bitwarden-production
only:
- apps_production
artifacts:
paths:
- apps/region2/*.xml
script:
- pod=$(kubectl get pods -n bitwarden-production | grep "self-host-admin" | awk '{print $1}')
- echo $pod
- kubectl cp -n bitwarden-production $pod:/etc/bitwarden/core/aspnet-dataprotection apps/region2/
apps-backup-region1-aspnet-keys:
stage: apps-aspnet-key-match
image: $K8S_DEPLOY_IMAGE
tags:
- region1-bitwarden-production
only:
- apps_production
artifacts:
paths:
- apps/region1/*.xml
script:
- pod=$(kubectl get pods -n bitwarden-production | grep "self-host-admin" | awk '{print $1}')
- echo $pod
- kubectl cp -n bitwarden-production $pod:/etc/bitwarden/core/aspnet-dataprotection apps/region1/
apps-copy-region1-aspnet-keys-to-region2:
stage: apps-aspnet-key-match
image: $K8S_DEPLOY_IMAGE
tags:
- region2-bitwarden-production
only:
- apps_production
artifacts:
paths:
- apps/region1/*.xml
needs:
- job: apps-backup-region1-aspnet-keys
script:
- pod=$(kubectl get pods -n bitwarden-production | grep "self-host-admin" | awk '{print $1}')
- echo $pod
- echo "Copying files into the pv, moving them because kubectl cp sucks, and cleaning up the old folder"
- kubectl cp -n bitwarden-production apps/region1/ $pod:/etc/bitwarden/core/aspnet-dataprotection
- kubectl exec -n bitwarden-production $pod -- bash -c "mv /etc/bitwarden/core/aspnet-dataprotection/region1/*.xml /etc/bitwarden/core/aspnet-dataprotection/ && rmdir /etc/bitwarden/core/aspnet-dataprotection/region1"
- echo "rollout restarting deployments so they have the correct xml keys"
- kubectl rollout restart deployments -n bitwarden-production
- while [[ ! $(curl -k -o /dev/null -s -w "%{http_code}\n" http://bitwarden-production-self-host-web.bitwarden-production:5000 -f) ]]; do sleep 2; done
apps-copy-region2-aspnet-keys-to-region1:
stage: apps-aspnet-key-match
image: $K8S_DEPLOY_IMAGE
tags:
- region1-bitwarden-production
only:
- apps_production
artifacts:
paths:
- apps/region2/*.xml
needs:
- job: apps-backup-region2-aspnet-keys
- job: apps-copy-region1-aspnet-keys-to-region2
script:
- pod=$(kubectl get pods -n bitwarden-production | grep "self-host-admin" | awk '{print $1}')
- echo $pod
- echo "Copying files into the pv, moving them because kubectl cp sucks, and cleaning up the old folder"
- kubectl cp -n bitwarden-production apps/region2/ $pod:/etc/bitwarden/core/aspnet-dataprotection
- kubectl exec -n bitwarden-production $pod -- bash -c "mv /etc/bitwarden/core/aspnet-dataprotection/region2/*.xml /etc/bitwarden/core/aspnet-dataprotection/ && rmdir /etc/bitwarden/core/aspnet-dataprotection/region2"
- echo "rollout restarting deployments so they have the correct xml keys"
- kubectl rollout restart deployments -n bitwarden-production
- while [[ ! $(curl -k -o /dev/null -s -w "%{http_code}\n" http://bitwarden-production-self-host-web.bitwarden-production:5000 -f) ]]; do sleep 2; done