nebari-dev
add service account with allow-app-sharing-role permissions
Reference Issues or PRs
In order for startup apps to be used by nebari, we need to create a service account with appropriate permissions to create the apps and share them with others. One issue this caused was the auth state for the service account doesn't get populated and it is needed in our custom Spawner code. This PR updates the service account auth_state during the pre-spawn-hook.
What does this implement/fix?
Put a x in the boxes that apply
- [ ] Bug fix (non-breaking change which fixes an issue)
- [x] New feature (non-breaking change which adds a feature)
- [ ] Breaking change (fix or feature that would cause existing features not to work as expected)
- [ ] Documentation Update
- [ ] Code style update (formatting, renaming)
- [ ] Refactoring (no functional changes, no API changes)
- [ ] Build related changes
- [ ] Other (please describe):
Testing
- [ ] Did you test the pull request locally?
- [ ] Did you add new tests?
How to test this PR?
- Do a nebari deployment with the following config defined in the nebari config file.
jhub_apps:
enabled: true
overrides: {
startup_apps: [
{
"username": "service-account-jupyterhub", # app will be created by this user
"servername": "my-startup-server", # specify a unique server name
"user_options": {
"display_name": "My Startup Server",
"description": "description",
"thumbnail": "data:image/jpeg;base64,iVBORw0KGgoAAAANSUhEUgAAAAoAAAAKCAYAAACNMs+9AAAAFUlEQVR42mNkYPhfz0AEYBxVSF+FAP5FDvcfRYWgAAAAAElFTkSuQmCC", # base64 encoded image data to use for thumbnail
"filepath": "panel_basic.py", # local file or path within git repo
"framework": "panel",
"public": False, # Whether or not app is publicly accessible without authentication
"keep_alive": False, # Whether or not to shut down app after a period of idleness
"env": {"MY_ENV_VAR": "MY_VALUE"},
"repository": {"url": "https://github.com/nebari-dev/jhub-apps-from-git-repo-example.git"}, # specify if pulling app from git repo
"conda_env": "global-panelenv",
"profile": "small-instance",
"share_with": {
"users": [],
"groups": ["/admin"]
},
},
},
]
}
- Then create this conda env after deployment in the
globalnamespace
name: panelenv
channels:
- conda-forge
dependencies:
- panel
- jhsingle-native-proxy
- bokeh-root-cmd
- ipykernel
variables: {}
- Then create an admin user and give the user the jupyterhub client's "allow-app-sharing-role" role. Log in and make sure you can see my-startup-server listed as a shared app. Then open it and ensure it opens correctly.
Any other comments?
I'm seeing some issues right now. The issue is the "jhub-apps-sa" user which creates the startup_apps needs to have logged in before the app server can be started successfully. In the current design, the "jhub-apps-sa" user is meant to act more like a service account and not log in interactively as users do. The issues I've seen so far are in the code that is run by the Spawner to set preferred username and render profiles, but it's possible there are more. In those instances, the auth state for the "jhub-apps-sa" user is None before initial log in so an error is thrown once we try to access info in the auth_state object in those methods.
Some possible ideas on how to fix this:
- if the auth_state is None, then go ask keycloak for the needed info? (login on behalf of the user maybe or some other way?)
- log the "jhub-apps-sa" user in somehow on startup app creation?
- I could add
startup_app: Trueto the user options dict used by jhub_apps. I could then modify our problematic spawner code which requires auth_state to only run if the server starting up isn't a startup app.
@krassowski any thoughts on how best to do this?
This might be too radical and not the kind of suggestion you are asking for, but could it be solved on the jhub-apps level? I mean jhub-apps is meant to be auth-provider agnostic so it should be possible to make this work without touching keycloak at all? This might be just my PTSD from figuring out keycloak piping last time speaking. I recall @aktech also had pleasure to work on keycloak integration - he may have better ideas.
Some possible ideas on how to fix this:
- if the auth_state is None, then go ask keycloak for the needed info? (login on behalf of the user maybe or some other way?)
- log the "jhub-apps-sa" user in somehow on startup app creation?
- I could add startup_app: True to the user options dict used by jhub_apps. I could then modify our problematic spawner code which requires
auth_stateto only run if the server starting up isn't a startup app.
If staying with this approach I would probably try the solutions in that exact order. I am not sure if the last one will work if you need to do anything beyond the configuration being set (i.e. whether server will actually spawn if you do not have auth_state).
This might be too radical and not the kind of suggestion you are asking for, but could it be solved on the jhub-apps level? I mean jhub-apps is meant to be auth-provider agnostic so it should be possible to make this work without touching keycloak at all?
That would be ideal, indeed. The way we are using the spawner here, expects the user to be logged in (hence populating auth_state) once before creating a server, which makes sense from jupyterhub pov as the servers are created by humans instead of robots.
This (startup apps) works in jhub-apps (without nebari) by default unless the spawner needs auth_state (which is the case here). Since its a feature of jhub-apps to provide the ability to create init apps on startup regardless of Authenticator or spawner, this should be handled in jhub-apps if possible (this is a big if though), you might need to pass in some kind of keycloak auth details in the JApps config, that might make it possible.
We can try to see if we can somehow call the authenticate method of the Authenticator in jhub-apps for the user, to populate the auth state before starting startup apps, but yes this might not be possible at all, in that case what you suggested in this comment https://github.com/nebari-dev/nebari/pull/2917#issuecomment-2607625067, sounds like a good approach, and I agree with @krassowski the last one won't work, as you need groups info to create nfs mounts.
Also, another thing to note here, is supporting this in jhub-apps might be tricky (if its possible), its probably ok to just go for your approach and we can tackle it in jhub-apps later, if time is of essence here.
I think the ideal solution is to create the startup apps up using jupyterhub-service-account which is service account associated with the jupyterhub Keycloak client. We already do the authenticatation needed in Nebari's KeycloakOAuthenticator. We just need to set auth_state for that service account after authentication. I'm not clear on how to set auth state for jupyterhub-service-account, but I'll dig in to the jupyterhub and jupyterhub/OAuthentication code further.
import json
import urllib.parse
import requests
import jwt
# def get_token():
client_id = "jupyterhub"
client_secret = "<my-secret>"
token_url = "https://github-actions.nebari.dev/auth/realms/nebari/protocol/openid-connect/token"
body = urllib.parse.urlencode({
"client_id": client_id,
"client_secret": client_secret,
"grant_type": "client_credentials",
})
headers = {
'Content-Type': 'application/x-www-form-urlencoded'
}
response = requests.post(token_url, data=body, headers=headers, verify=False)
data = response.json()
# Get the token
token = data["access_token"]
# Decode and print token contents
decoded = jwt.decode(token, options={"verify_signature": False})
print(json.dumps(decoded, indent=2))
yields
{
"exp": 1737998505,
"iat": 1737998205,
"jti": "9827b1e3-9612-4341-8076-22e99d2ccb04",
"iss": "https://github-actions.nebari.dev/auth/realms/nebari",
"aud": [
"realm-management",
"grafana",
"argo-server-sso",
"conda_store",
"account"
],
"sub": "c1da7cbd-3150-42ae-8b19-c9b4304f2054",
"typ": "Bearer",
"azp": "jupyterhub",
"acr": "1",
"realm_access": {
"roles": [
"offline_access",
"default-roles-nebari",
"uma_authorization"
]
},
"resource_access": {
"realm-management": {
"roles": [
"view-realm",
"view-users",
"view-clients",
"query-clients",
"query-groups",
"query-users"
]
},
"jupyterhub": {
"roles": [
"allow-read-access-to-services-role",
"jupyterhub_developer",
"allow-group-directory-creation-role"
]
},
"grafana": {
"roles": [
"grafana_viewer"
]
},
"argo-server-sso": {
"roles": [
"argo-viewer"
]
},
"conda_store": {
"roles": [
"conda_store_developer"
]
},
"account": {
"roles": [
"manage-account",
"manage-account-links",
"view-profile"
]
}
},
"scope": "email profile",
"clientHost": "10.244.0.1",
"email_verified": false,
"clientId": "jupyterhub",
"roles": [
"view-realm",
"view-users",
"view-clients",
"query-clients",
"query-groups",
"query-users",
"allow-read-access-to-services-role",
"jupyterhub_developer",
"allow-group-directory-creation-role",
"grafana_viewer",
"argo-viewer",
"conda_store_developer",
"manage-account",
"manage-account-links",
"view-profile"
],
"groups": [
"/analyst",
"/users"
],
"preferred_username": "service-account-jupyterhub",
"clientAddress": "10.244.0.1"
}
which has the group membership for the service account, preferred_username, and permissions similar to a normal user.
~~I haven't been able to successfully set auth state for the service account,~~ (Update: resolved by commit 110b0ee (next commit)) but just looking up the service account's info during the spawner code in this commit seems to work.
I was originally planning on adding a test once I had a ragna distribution running that would test this, but I'll add a test for just this part (startup_apps).
Looks good overall, I haven't had the chance to test it locally due to M1 issues, but will try to sort that out tomorrow.
How do you feel about adding a test here verifying the service account has correct roles?
@aktech, I added this test now in 80456c5, but it failed since test-user is not an admin. The test uses his jupyterhub token so I make the test-user an admin and it now works as expected.
Looks like its still failing: https://github.com/nebari-dev/nebari/actions/runs/13252428164/job/36993020527#step:13:220 (?)
Looks like its still failing: nebari-dev/nebari/actions/runs/13252428164/job/36993020527#step:13:220 (?)
Thanks for pointing that out, @aktech. Yeah, this won't pass until we do a jhub_apps release and include the latest in the jupyterhub image used by nebari. I opened an issue about the jhub-apps release. I'll convert to draft for now.
Amit reminded me that the startup app expects a global-mypanel env which we aren't creating and I'll need to try to switch it over to the nebari-git/dashboards one or build the global-mypanel env if we want to actually try to open the startup server.
Amit reminded me that the startup app expects a global-mypanel env which we aren't creating and I'll need to try to switch it over to the nebari-git/dashboards one or build the global-mypanel env if we want to actually try to open the startup server.
I am curious on how that will work, let me know if you need any help.
Amit reminded me that the startup app expects a global-mypanel env which we aren't creating and I'll need to try to switch it over to the nebari-git/dashboards one or build the global-mypanel env if we want to actually try to open the startup server.
I am curious on how that will work, let me know if you need any help.
According to @aktech, we would need to add:
- jhsingle-native-proxy
- bokeh-root-cmd to the dashboard environment to match the app requirements.
so it should be straight forward.
Out of date, and not high priority, though good useful work. I'll close for now