zero-to-jupyterhub-k8s
zero-to-jupyterhub-k8s copied to clipboard
jupyterhub
cull_idle not being able to clean up because of HTTP: 403 Forbidden
Bug description
Even though my chart has values:
cull:
enabled: true
user hubs are not cleaned up. I am on chart v4.0.0, azure deployment. The problem is that culler can't seem to get to the hub API:
│ [I 2025-01-20 20:15:13.029 JupyterHub log:192] 200 POST /hub/api/users/jackwildman/activity ([email protected]) 15.16ms │
│ [I 2025-01-20 20:16:20.171 JupyterHub log:192] 200 GET /hub/api/ (jupyterhub-idle-culler@::1) 10.61ms │
│ [W 2025-01-20 20:16:20.174 JupyterHub scopes:1019] Not authorizing access to /hub/api/users. Requires any of [list:users] on *, not derived from scopes [] │
│ [W 2025-01-20 20:16:20.174 JupyterHub web:1873] 403 GET /hub/api/users?state=ready (::1): Action is not authorized with current scopes; requires any of [list:users]
│ [W 2025-01-20 20:16:20.174 JupyterHub log:192] 403 GET /hub/api/users?state=[secret] (jupyterhub-idle-culler@::1) 1.65ms │
│ [E 250120 20:16:20 ioloop:941] Exception in callback functools.partial(<function cull_idle at 0x7f41a330b6a0>, url='http://localhost:8081/hub/api', api_token='89f1f94b53c643bb8abbbbb4f9b27180bd87ead', inactive_limit=3600, cull_users=False, remove_named_servers=False, max_age=0, concurrency=10, ssl_enabled=False, internal_certs_location='internal-ssl', cull_admin_users=True │
│ , api_page_size=0, cull_default_servers=True, cull_named_servers=True) │
│ Traceback (most recent call last): │
│ File "/usr/local/lib/python3.12/site-packages/tornado/ioloop.py", line 939, in _run │
│ await val │
│ File "/usr/local/lib/python3.12/site-packages/jupyterhub_idle_culler/__init__.py", line 436, in cull_idle │
│ async for user in fetch_paginated(req): │
│ File "/usr/local/lib/python3.12/site-packages/jupyterhub_idle_culler/__init__.py", line 142, in fetch_paginated │
│ response = await resp_future │
│ ^^^^^^^^^^^^^^^^^ │
│ File "/usr/local/lib/python3.12/site-packages/jupyterhub_idle_culler/__init__.py", line 124, in fetch │
│ return await client.fetch(req) │
│ ^^^^^^^^^^^^^^^^^^^^^^^ │
│ tornado.httpclient.HTTPClientError: HTTP 403: Forbidden │
│ [I 2025-01-20 20:16:33.478 JupyterHub log:192] 200 GET /hub/api/user ([email protected]) 10.44ms
Sure:
# This file can update the JupyterHub Helm chart's default configuration values.
#
# For reference see the configuration reference and default values, but make
# sure to refer to the Helm chart version of interest to you!
#
# Introduction to YAML: https://www.youtube.com/watch?v=cdLNKUoMc6c
# Chart config reference: https://zero-to-jupyterhub.readthedocs.io/en/stable/resources/reference.html
# Chart default values: https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/HEAD/jupyterhub/values.yaml
# Available chart versions: https://hub.jupyter.org/helm-chart/
#
proxy:
# we don't need a separate loadbalancer, we will use Traefik
service:
type: ClusterIP
disableHttpPort: false
https:
enabled: false
ingress:
enabled: false
hub:
config:
Authenticator:
admin_users:
- ****
GitHubOAuthenticator:
client_id: ****
client_secret: ****
oauth_callback_url: https://****/hub/oauth_callback
allowed_organizations:
- ****
enable_auth_state: true
allow_all: true
JupyterHub:
authenticator_class: github
load_roles:
- name: user
description: Ovewriting default to be able to share access to your servers
scopes:
- self
# as per https://jupyterhub.readthedocs.io/en/stable/reference/sharing.html
- shares!user
- read:users:name
- read:groups:name
# these are for being able to access the admin ui https://discourse.jupyter.org/t/real-time-collaboration-z2jh/24059/6
# could be separated in their own groups
- access:servers
- admin:servers
- admin-ui
- list:users
singleuser:
# Defines the default image
image:
name: us-central1-docker.pkg.dev/***/jupyterhub-user-image/jupyter-alpha
tag: "0.0.6"
pullSecrets:
- name: gcp-us-central1-secret
profileList:
- display_name: "Default Python Data Science Env"
description: "**** custom data analysis stack with some extra extensions"
default: true
kubespawner_override:
lifecycle_hooks:
postStart:
exec:
command:
- "sh"
- "-c"
- >
cp /tmp/StartHere.ipynb /home/jovyan/ &&
cp -r /tmp/.condarc /home/jovyan/.condarc &&
cp -r /tmp/.ssh /home/jovyan/.ssh &&
chmod 600 /home/jovyan/.ssh/id_rsa &&
[ ! -d /home/jovyan/research ] && cp -r /tmp/research /home/jovyan/research || true
- display_name: "Minimal default scipy notebook"
description: "Minimal default scipy notebook environment"
kubespawner_override:
image: quay.io/jupyter/scipy-notebook
tag: latest
extraEnv:
EDITOR: "vim"
SERPER_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: SERPER_API_KEY
ANTHROPIC_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: ANTHROPIC_API_KEY
OPENAI_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: OPENAI_API_KEY
RETROSEARCH_URL:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: RETROSEARCH_URL
GEMINI_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: GEMINI_API_KEY
DAGSTER_HOME:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: DAGSTER_HOME
PYTHONPATH:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: PYTHONPATH
PYTHON_REPL_SERVER_PORT:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: PYTHON_REPL_SERVER_PORT
PERPLEXITYAI_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: PERPLEXITYAI_API_KEY
LANGSMITH_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGSMITH_API_KEY
LANGSMITH_PROJECT:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGSMITH_PROJECT
LANGSMITH_DEFAULT_RUN_NAME:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGSMITH_DEFAULT_RUN_NAME
LANGCHAIN_TRACING_V2:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGCHAIN_TRACING_V2
LANGCHAIN_ENDPOINT:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGCHAIN_ENDPOINT
LANGCHAIN_API_KEY:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGCHAIN_API_KEY
LANGCHAIN_PROJECT:
valueFrom:
secretKeyRef:
name: jupyter-user-secret
key: LANGCHAIN_PROJECT
cull:
enabled: true
I don't know if you have solved your problem since this is an old issue but I had the exact same issue. It turned out I was overwritting c.JupyterHub.load_roles which removed jupyterhub-idle-culler role. You are doing exactly that with
hub:
config:
JupyterHub:
load_roles:
- ....
If you want to append to the list of roles you should use
hub:
loadRoles:
- .....
which gets evaluated in jupyterhub_config.py here https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/c8959795b1a079401034fbac1b63c82a6cb5cdbf/jupyterhub/files/hub/jupyterhub_config.py#L415
