systemdspawner icon indicating copy to clipboard operation
systemdspawner copied to clipboard

Published 20 hours ago • verified publisher icon jupyterhub

Unable to spawn user instances on CentOS 7 with v0.15.0

Open astro-arphid opened this issue 3 years ago • 6 comments

Bug description

Unable to spawn user sessions using jupyterhub-systemdspawner==0.15.0.

I noticed that 0.14.0 is the latest listed under releases on the repository, so downgraded to that, which allows user instances to spawn with the same parameters configured.

I believe that this entry from the logs below is part of the problem, which was resolved under #9.

Feb 22 15:34:29 sysdspawner scl[3485]: Unknown assignment RuntimeDirectory=jupyter-test-singleuser.
Feb 22 15:34:29 sysdspawner scl[3485]: Failed to create bus message: No such device or address

Expected behaviour

User sessions should spawn instead of timing out.

Actual behaviour

Users see the below when logging in:

Spawn failed: Timeout
Event log
Server requested
Spawning server...
Spawn failed: Timeout

How to reproduce

  1. Log into JupyterHub using known good credentials
  2. See above error

Your personal set up

  • OS:
# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
  • Software:
# systemctl --version
systemd 219

# uname -r
3.10.0-1160.15.2.el7.x86_64

# python --version
Python 3.8.6

# pip list
jupyterhub                1.3.0
jupyterhub-systemdspawner 0.15.0
jupyterlab                2.2.5
  • Full environment 
# pip freeze
alembic==1.5.4
anyio==2.1.0
appdirs==1.4.4
argon2-cffi==20.1.0
async-generator==1.10
attrs==20.3.0
Babel==2.9.0
backcall==0.2.0
bleach==3.3.0
certifi==2020.12.5
certipy==0.1.3
cffi==1.14.4
chardet==4.0.0
cryptography==3.4.4
decorator==4.4.2
defusedxml==0.6.0
distlib==0.3.1
entrypoints==0.3
filelock==3.0.12
idna==2.10
ipykernel==5.4.3
ipython==7.20.0
ipython-genutils==0.2.0
jedi==0.18.0
Jinja2==2.11.3
json5==0.9.5
jsonschema==3.2.0
jupyter-client==6.1.11
jupyter-core==4.7.1
jupyter-server==1.3.0
jupyter-telemetry==0.1.0
jupyterhub==1.3.0
jupyterhub-systemdspawner==0.15.0
jupyterlab==2.2.5
jupyterlab-pygments==0.1.2
jupyterlab-server==1.2.0
Mako==1.1.4
MarkupSafe==1.1.1
mistune==0.8.4
nbclassic==0.2.6
nbclient==0.5.2
nbconvert==6.0.7
nbformat==5.1.2
nest-asyncio==1.5.1
notebook==6.2.0
oauthlib==3.1.0
packaging==20.9
pamela==1.0.0
pandocfilters==1.4.3
parso==0.8.1
pexpect==4.8.0
pickleshare==0.7.5
prometheus-client==0.9.0
prompt-toolkit==3.0.14
ptyprocess==0.7.0
pycparser==2.20
Pygments==2.7.4
pyOpenSSL==20.0.1
pyparsing==2.4.7
pyrsistent==0.17.3
python-dateutil==2.8.1
python-editor==1.0.4
python-json-logger==2.0.1
pytz==2021.1
pyzmq==22.0.2
requests==2.25.1
ruamel.yaml==0.16.12
ruamel.yaml.clib==0.2.2
Send2Trash==1.5.0
six==1.15.0
sniffio==1.2.0
SQLAlchemy==1.3.23
terminado==0.9.2
testpath==0.4.4
tornado==6.1
traitlets==5.0.5
urllib3==1.26.3
virtualenv==20.0.31
wcwidth==0.2.5
webencodings==0.5.1
  • Configuration 
# jupyterhub_config.py
import sys
import os

c.JupyterHub.cookie_secret_file = 'jupyterhub_cookie_secret'
c.JupyterHub.db_url = 'sqlite:///jupyterhub.sqlite'
c.ConfigurableHTTPProxy.pid_file = "jupyterhub-proxy.pid"

c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
c.JupyterHub.proxy_class = 'jupyterhub.proxy.ConfigurableHTTPProxy'

#c.Spawner.cmd = ['jupyterhub-singleuser']
c.Spawner.cmd = ['jupyter-labhub']
c.SystemdSpawner.default_url = '/lab?reset'

c.PAMAuthenticator.open_sessions = False
c.SystemdSpawner.mem_limit = '1G'
c.SystemdSpawner.cpu_limit = 1.0
c.SystemdSpawner.default_shell = '/bin/bash'
c.SystemdSpawner.user_workingdir = '/home/{USERNAME}'

# Requires more recent systemd
#c.SystemdSpawner.isolate_tmp = True
#c.SystemdSpawner.disable_user_sudo = True
#c.SystemdSpawner.dynamic_users = False
#c.SystemdSpawner.isolate_devices = True
#c.SystemdSpawner.readonly_paths = ['/']
#c.SystemdSpawner.readwrite_paths = ['/home/{USERNAME}', '/var/log/']
#c.SystemdSpawner.username_template = '{USERNAME}'

# Deprecated
c.JupyterHub.extra_log_file = "/var/log/jupyterhub.log"

c.JupyterHub.log_level = "DEBUG"
c.Spawner.debug = True
c.SystemdSpawner.debug = True
c.Spawner.start_timeout = 30
#c.JupyterHub.init_spawners_timeout = 30
c.ConfigurableHTTPProxy.debug = True
  • Logs

journalctl

Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.461 JupyterHub base:525] Setting cookie for test: jupyterhub-hub-login
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.461 JupyterHub base:521] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
Feb 22 15:34:29 sysdspawner scl[3485]: [I 2021-02-22 15:34:29.461 JupyterHub base:757] User logged in: test
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.461 JupyterHub user:288] Creating <class 'systemdspawner.systemdspawner.SystemdSpawner'> for test:
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.463 JupyterHub systemdspawner:155] user:test Initialized spawner with unit jupyter-test-singleuser
Feb 22 15:34:29 sysdspawner scl[3485]: [I 2021-02-22 15:34:29.463 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn (test@::ffff:192.168.56.1) 369.89ms
Feb 22 15:34:29 sysdspawner scl[3485]: 15:34:29.483 [ConfigProxy] debug: PROXY WEB /hub/spawn to http://127.0.0.1:8081
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.487 JupyterHub pages:217] Triggering spawn with default options for test
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.487 JupyterHub base:875] Initiating spawn for test
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.487 JupyterHub base:879] 0/100 concurrent spawns
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.487 JupyterHub base:884] 0 active servers
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.511 JupyterHub user:602] Calling Spawner.start for test
Feb 22 15:34:29 sysdspawner scl[3485]: [D 2021-02-22 15:34:29.512 JupyterHub systemdspawner:201] user:test Using port 56903 to start spawning user server
Feb 22 15:34:29 sysdspawner scl[3485]: Unknown assignment RuntimeDirectory=jupyter-test-singleuser.
Feb 22 15:34:29 sysdspawner scl[3485]: Failed to create bus message: No such device or address
Feb 22 15:34:30 sysdspawner scl[3485]: [I 2021-02-22 15:34:30.494 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/test (test@::ffff:192.168.56.1) 1008.57ms
Feb 22 15:34:30 sysdspawner scl[3485]: 15:34:30.544 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/test to http://127.0.0.1:8081
Feb 22 15:34:30 sysdspawner scl[3485]: [I 2021-02-22 15:34:30.557 JupyterHub pages:402] test is pending spawn
Feb 22 15:34:30 sysdspawner scl[3485]: [I 2021-02-22 15:34:30.561 JupyterHub log:181] 200 GET /hub/spawn-pending/test (test@::ffff:192.168.56.1) 8.44ms
Feb 22 15:34:30 sysdspawner scl[3485]: 15:34:30.677 [ConfigProxy] debug: PROXY WEB /hub/api/users/test/server/progress to http://127.0.0.1:8081
Feb 22 15:34:39 sysdspawner scl[3485]: [W 2021-02-22 15:34:39.490 JupyterHub base:1003] User test is slow to start (timeout=10)
Feb 22 15:34:59 sysdspawner scl[3485]: [W 2021-02-22 15:34:59.515 JupyterHub user:681] test's server failed to start in 30 seconds, giving up
Feb 22 15:34:59 sysdspawner scl[3485]: [D 2021-02-22 15:34:59.517 JupyterHub user:790] Stopping test
Feb 22 15:34:59 sysdspawner scl[3485]: [D 2021-02-22 15:34:59.556 JupyterHub user:816] Deleting oauth client jupyterhub-user-test
Feb 22 15:34:59 sysdspawner scl[3485]: [D 2021-02-22 15:34:59.563 JupyterHub user:819] Finished stopping test
Feb 22 15:34:59 sysdspawner scl[3485]: [E 2021-02-22 15:34:59.569 JupyterHub gen:623] Exception in Future <Task finished name='Task-23' coro=<BaseHandler.spawn_single_user.<locals>.fini
Feb 22 15:34:59 sysdspawner scl[3485]: Traceback (most recent call last):
Feb 22 15:34:59 sysdspawner scl[3485]: File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/tornado/gen.py", line 618, in error_callback
Feb 22 15:34:59 sysdspawner scl[3485]: future.result()
Feb 22 15:34:59 sysdspawner scl[3485]: File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 902, in finish_user_spawn
Feb 22 15:34:59 sysdspawner scl[3485]: await spawn_future
Feb 22 15:34:59 sysdspawner scl[3485]: File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 707, in spawn
Feb 22 15:34:59 sysdspawner scl[3485]: raise e
Feb 22 15:34:59 sysdspawner scl[3485]: File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 606, in spawn
Feb 22 15:34:59 sysdspawner scl[3485]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
Feb 22 15:34:59 sysdspawner scl[3485]: tornado.util.TimeoutError: Timeout
Feb 22 15:34:59 sysdspawner scl[3485]: [I 2021-02-22 15:34:59.570 JupyterHub log:181] 200 GET /hub/api/users/test/server/progress (test@::ffff:192.168.56.1) 28891.96ms

/var/log/jupyterhub.log

[I 2021-02-22 15:34:21.166 JupyterHub app:2349] Running JupyterHub version 1.3.0
[I 2021-02-22 15:34:21.167 JupyterHub app:2379] Using Authenticator: jupyterhub.auth.PAMAuthenticator-1.3.0
[I 2021-02-22 15:34:21.167 JupyterHub app:2379] Using Spawner: systemdspawner.systemdspawner.SystemdSpawner
[I 2021-02-22 15:34:21.167 JupyterHub app:2379] Using Proxy: jupyterhub.proxy.ConfigurableHTTPProxy-1.3.0
[D 2021-02-22 15:34:21.168 JupyterHub app:2310] Could not load pycurl: No module named 'pycurl'
    pycurl is recommended if you have a large number of users.
[I 2021-02-22 15:34:21.168 JupyterHub app:1420] Loading cookie_secret from /srv/jupyterhub/jupyterhub_cookie_secret
[D 2021-02-22 15:34:21.168 JupyterHub app:1587] Connecting to db: sqlite:///jupyterhub.sqlite
[D 2021-02-22 15:34:21.181 JupyterHub orm:815] database schema version found: 4dc2d5a8c53c
[I 2021-02-22 15:34:21.184 JupyterHub proxy:460] Generating new CONFIGPROXY_AUTH_TOKEN
[W 2021-02-22 15:34:21.185 JupyterHub app:1695] No admin users, admin interface will be unavailable.
[W 2021-02-22 15:34:21.185 JupyterHub app:1696] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2021-02-22 15:34:21.185 JupyterHub app:1725] Not using allowed_users. Any authenticated user will be allowed.
[D 2021-02-22 15:34:21.204 JupyterHub app:1877] Purging expired APITokens
[D 2021-02-22 15:34:21.206 JupyterHub app:1877] Purging expired OAuthAccessTokens
[D 2021-02-22 15:34:21.209 JupyterHub app:1877] Purging expired OAuthCodes
[D 2021-02-22 15:34:21.215 JupyterHub app:2004] Initializing spawners
[D 2021-02-22 15:34:21.216 JupyterHub app:2137] Loaded users:
    
[I 2021-02-22 15:34:21.216 JupyterHub app:2416] Initialized 0 spawners in 0.001 seconds
[W 2021-02-22 15:34:21.218 JupyterHub proxy:662] Running JupyterHub without SSL.  I hope there is SSL termination happening somewhere else...
[I 2021-02-22 15:34:21.218 JupyterHub proxy:666] Starting proxy @ http://:8000
[D 2021-02-22 15:34:21.218 JupyterHub proxy:667] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--error-target', 'h
ttp://127.0.0.1:8081/hub/error', '--log-level', 'debug']
[D 2021-02-22 15:34:21.224 JupyterHub proxy:574] Writing proxy pid file: jupyterhub-proxy.pid
[D 2021-02-22 15:34:21.522 JupyterHub proxy:702] Proxy started and appears to be up
[D 2021-02-22 15:34:21.526 JupyterHub proxy:795] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2021-02-22 15:34:21.539 JupyterHub app:2664] Hub API listening on http://127.0.0.1:8081/hub/
[D 2021-02-22 15:34:21.539 JupyterHub proxy:314] Fetching routes to check
[D 2021-02-22 15:34:21.539 JupyterHub proxy:795] Proxy: Fetching GET http://127.0.0.1:8001/api/routes
[I 2021-02-22 15:34:21.542 JupyterHub proxy:319] Checking routes
[I 2021-02-22 15:34:21.542 JupyterHub proxy:399] Adding default route for Hub: / => http://127.0.0.1:8081
[D 2021-02-22 15:34:21.542 JupyterHub proxy:795] Proxy: Fetching POST http://127.0.0.1:8001/api/routes/
[I 2021-02-22 15:34:21.547 JupyterHub app:2739] JupyterHub is now running at http://:8000
[D 2021-02-22 15:34:21.548 JupyterHub app:2342] It took 0.398 seconds for the Hub to start
[D 2021-02-22 15:34:25.681 JupyterHub base:328] Refreshing auth for test
[I 2021-02-22 15:34:25.681 JupyterHub login:43] User logged out: test
[I 2021-02-22 15:34:25.685 JupyterHub log:181] 302 GET /hub/logout -> /hub/login (@::ffff:192.168.56.1) 14.43ms
[I 2021-02-22 15:34:25.729 JupyterHub log:181] 200 GET /hub/login (@::ffff:192.168.56.1) 25.28ms
[D 2021-02-22 15:34:29.461 JupyterHub base:521] Setting cookie jupyterhub-session-id: {'httponly': True}
[D 2021-02-22 15:34:29.461 JupyterHub base:525] Setting cookie for test: jupyterhub-hub-login
[D 2021-02-22 15:34:29.461 JupyterHub base:521] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'}
[I 2021-02-22 15:34:29.461 JupyterHub base:757] User logged in: test
[D 2021-02-22 15:34:29.461 JupyterHub user:288] Creating <class 'systemdspawner.systemdspawner.SystemdSpawner'> for test:
[D 2021-02-22 15:34:29.463 JupyterHub systemdspawner:155] user:test Initialized spawner with unit jupyter-test-singleuser
[I 2021-02-22 15:34:29.463 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn (test@::ffff:192.168.56.1) 369.89ms
[D 2021-02-22 15:34:29.487 JupyterHub pages:217] Triggering spawn with default options for test
[D 2021-02-22 15:34:29.487 JupyterHub base:875] Initiating spawn for test
[D 2021-02-22 15:34:29.487 JupyterHub base:879] 0/100 concurrent spawns
[D 2021-02-22 15:34:29.487 JupyterHub base:884] 0 active servers
[D 2021-02-22 15:34:29.511 JupyterHub user:602] Calling Spawner.start for test
[D 2021-02-22 15:34:29.512 JupyterHub systemdspawner:201] user:test Using port 56903 to start spawning user server
[I 2021-02-22 15:34:30.494 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/test (test@::ffff:192.168.56.1) 1008.57ms
[I 2021-02-22 15:34:30.557 JupyterHub pages:402] test is pending spawn
[I 2021-02-22 15:34:30.561 JupyterHub log:181] 200 GET /hub/spawn-pending/test (test@::ffff:192.168.56.1) 8.44ms
[W 2021-02-22 15:34:39.490 JupyterHub base:1003] User test is slow to start (timeout=10)
[W 2021-02-22 15:34:59.515 JupyterHub user:681] test's server failed to start in 30 seconds, giving up
[D 2021-02-22 15:34:59.517 JupyterHub user:790] Stopping test
[D 2021-02-22 15:34:59.556 JupyterHub user:816] Deleting oauth client jupyterhub-user-test
[D 2021-02-22 15:34:59.563 JupyterHub user:819] Finished stopping test
[E 2021-02-22 15:34:59.569 JupyterHub gen:623] Exception in Future <Task finished name='Task-23' coro=<BaseHandler.spawn_single_user.<locals>.finish_user_spawn() done, defined at /opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py:895> exception=TimeoutError('Timeout')> after timeout
    Traceback (most recent call last):
      File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/tornado/gen.py", line 618, in error_callback
        future.result()
      File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 902, in finish_user_spawn
        await spawn_future
      File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 707, in spawn
        raise e
      File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 606, in spawn
        url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
    tornado.util.TimeoutError: Timeout
    
[I 2021-02-22 15:34:59.570 JupyterHub log:181] 200 GET /hub/api/users/test/server/progress (test@::ffff:192.168.56.1) 28891.96ms
(END)

astro-arphid avatar Feb 22 '21 17:02 astro-arphid

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively. welcome You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:

welcome[bot] avatar Feb 22 '21 17:02 welcome[bot]

I recommend strongly against downgrading to 0.14. The issue is with 1b83c2738f4a3fa34b5242da9cb098a04901e6dc which fixes GHSA-cg54-gpgr-4rm6, which, as can be seen in the log, uses RuntimeDirectory=, that was only introduced in systemd v228.

I know the appeal of CentOS is to get updates until the end of time (or nowadays until the end of this year), but using software this outdated is somewhat problematic. Users of CentOS at scale do backport some packages to keep it usable, I recommend having a look at Facebook's rpm-backports

behrmann avatar Feb 22 '21 17:02 behrmann

@behrmann Appreciate the advice, thanks! We're in a bit of a tight position at the moment. Plans were to upgrade, but recent events with CentOS have, like with many other orgs, raised cause for concern and put a halt on that while we figure out what the best path is going forward.

Unfortunately downgrading was the only way I could get this working, and now I see why. I'm not going live with this yet as ideally would like to be running the most recent supported version. systemdspawner is the least complex option for implementing resource restrictions against JupyterHub users, something that wasn't so much of an issue pre-pandemic, but now that everyone is working from home and most users don't have the functionality to run Jupyter*... we're a bit pushed.

I'll take a look. Thanks!

astro-arphid avatar Feb 22 '21 17:02 astro-arphid

I can confirm this issue.

My system

  • OpenSUSE Leap 15.2, kernel 5.3.18-lp152.63-default
  • Python 3.6.12
  • jupyterhub 1.3.0
  • jupyterhub-systemdspawner 0.15.0
  • systemd 234

Also, downgrading to 0.14 solves the issue.

journalctl

Mar 02 19:52:49 jh jupyterhub[2280]: [I 2021-03-02 19:52:49.903 JupyterHub base:757] User logged in: tux Mar 02 19:52:49 jh jupyterhub[2280]: [I 2021-03-02 19:52:49.906 JupyterHub log:181] 302 POST /hub/login?next= -> /hub/spawn ([email protected]) 28.38ms Mar 02 19:52:50 jh jupyterhub[2280]: Unknown assignment RuntimeDirectoryMode=700. Mar 02 19:52:50 jh jupyterhub[2280]: Failed to create bus message: Invalid argument Mar 02 19:52:50 jh jupyterhub[2280]: [I 2021-03-02 19:52:50.989 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/tux ([email protected]) 1> Mar 02 19:52:51 jh jupyterhub[2280]: [I 2021-03-02 19:52:51.070 JupyterHub pages:402] tux is pending spawn Mar 02 19:52:51 jh jupyterhub[2280]: [I 2021-03-02 19:52:51.098 JupyterHub log:181] 200 GET /hub/spawn-pending/tux ([email protected]) 31.93ms Mar 02 19:52:59 jh jupyterhub[2280]: [W 2021-03-02 19:52:59.987 JupyterHub base:1006] User tux is slow to start (timeout=10) Mar 02 19:53:50 jh jupyterhub[2280]: [W 2021-03-02 19:53:50.032 JupyterHub user:683] tux's server failed to start in 60 seconds, giving up Mar 02 19:53:50 jh jupyterhub[2280]: [E 2021-03-02 19:53:50.078 JupyterHub gen:624] Exception in Future <Task finished coro=<BaseHandler.spawn_single_u> Mar 02 19:53:50 jh jupyterhub[2280]: Traceback (most recent call last): Mar 02 19:53:50 jh jupyterhub[2280]: File "/opt/jupyterhub/lib64/python3.6/site-packages/tornado/gen.py", line 618, in error_callback Mar 02 19:53:50 jh jupyterhub[2280]: future.result() Mar 02 19:53:50 jh jupyterhub[2280]: File "/opt/jupyterhub/lib64/python3.6/site-packages/jupyterhub/handlers/base.py", line 902, in finish_user_s> Mar 02 19:53:50 jh jupyterhub[2280]: await spawn_future Mar 02 19:53:50 jh jupyterhub[2280]: File "/opt/jupyterhub/lib64/python3.6/site-packages/jupyterhub/user.py", line 707, in spawn Mar 02 19:53:50 jh jupyterhub[2280]: raise e Mar 02 19:53:50 jh jupyterhub[2280]: File "/opt/jupyterhub/lib64/python3.6/site-packages/jupyterhub/user.py", line 606, in spawn Mar 02 19:53:50 jh jupyterhub[2280]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f) Mar 02 19:53:50 jh jupyterhub[2280]: tornado.util.TimeoutError: Timeout

pmvr avatar Mar 02 '21 19:03 pmvr

This also occurs on Ubuntu 16.04 LTS and other similarly old systemd versions. Luckily that reaches end of support next month, so not too much to do there.

mariusvniekerk avatar Mar 16 '21 04:03 mariusvniekerk

For openSUSE Leap 15.2 which has systemd version 234 I got it to work by commenting the following two lines:

https://github.com/jupyterhub/systemdspawner/blob/ee566d5989ca004bc6f4a3bf5a0a7cd332da2e90/systemdspawner/systemd.py#L98-L101

Now the folder is world-readable, however, the env file is not - so I guess that is fine?

drwxr-xr-x 2 mathause wheel /run/jupyter-mathause-singleuser
-r-------- 1 root     root  /run/jupyter-mathause-singleuser/jupyter-mathause-singleuser.env

It seems to have survived a restart but I have not thoroughly tested this.

mathause avatar Mar 28 '21 18:03 mathause