nebari icon indicating copy to clipboard operation
nebari copied to clipboard

[BUG] - Hub pod fails if keycloak doesn't starts before it

Open aktech opened this issue 1 year ago â€Ē 0 comments

Describe the bug

The hub pod Errors out if keycloak isn't started already.

│ [I 2024-06-27 10:02:20.759 JupyterHub app:3316] Using Authenticator: builtins.KeyCloakOAuthenticator                                                                                                                                                                         │
│ [I 2024-06-27 10:02:20.759 JupyterHub app:3316] Using Spawner: jhub_apps.spawner.spawner_creation.JHubSpawner                                                                                                                                                                │
│ [I 2024-06-27 10:02:20.759 JupyterHub app:3316] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-5.0.0                                                                                                                                                                    │
│ [I 2024-06-27 10:02:20.801 JupyterHub <string>:109] Loading managed roles                                                                                                                                                                                                    │
│ [E 2024-06-27 10:02:20.811 JupyterHub app:3852]                                                                                                                                                                                                                              │
│     Traceback (most recent call last):                                                                                                                                                                                                                                       │
│       File "/opt/conda/lib/python3.9/site-packages/jupyterhub/app.py", line 3849, in launch_instance_async                                                                                                                                                                   │
│         await self.initialize(argv)                                                                                                                                                                                                                                          │
│       File "/opt/conda/lib/python3.9/site-packages/jupyterhub/app.py", line 3332, in initialize                                                                                                                                                                              │
│         await self.init_role_creation()                                                                                                                                                                                                                                      │
│       File "/opt/conda/lib/python3.9/site-packages/jupyterhub/app.py", line 2286, in init_role_creation                                                                                                                                                                      │
│         managed_roles = await self.authenticator.load_managed_roles()                                                                                                                                                                                                        │
│       File "<string>", line 114, in load_managed_roles                                                                                                                                                                                                                       │
│       File "<string>", line 221, in _get_token                                                                                                                                                                                                                               │
│     tornado.httpclient.HTTPClientError: HTTP 503: Service Unavailable

This is due to the fact that jupyterhub calls keycloak API on startup to get roles into JupyterHub (see load_managed_roles)

Expected behavior

Hub waits (retries for some time) for keycloak to start before dying out.

OS and architecture in which you are running Nebari

NA

How to Reproduce the problem?

Kill keycloak pod and restart Hub pod

Command output

No response

Versions and dependencies used.

No response

Compute environment

None

Integrations

No response

Anything else?

No response

aktech avatar Jun 27 '24 10:06 aktech