Draft: backport fix for GHSA-cg54-gpgr-4rm6 to CentOS 7

[behrmann]

This is a reaction to #79

@astro-arphid This is just spitballing, but this is what a backport of the fix might look like. I haven't tested it, though. Would be grat to hear from you if this works.

[bodhar]

@behrmann Thanks for getting to this so quickly!

Putting this in place with version 0.15.0, I'm getting the following when logging in:

500 : Internal Server Error

Error in Authenticator.pre_spawn_start: UnboundLocalError local variable 'runtime_dir' referenced before assignment

You can try restarting your server from the home page.

From the logs:

Feb 22 19:18:31 sysdspawner scl[8557]: [E 2021-02-22 19:18:31.363 JupyterHub user:689] Unhandled error starting test's server: local variable 'runtime_dir' referenced before assignment

[behrmann]

Ah, I had overlooked the reuse of that variable. Moved some things around, might work now.

[bodhar]

Thanks! Timing out this time.

Feb 22 20:09:15 sysdspawner scl[9346]: [D 2021-02-22 20:09:15.778 JupyterHub systemdspawner:201] user:test Using port 34637 to start spawning user server
Feb 22 20:09:15 sysdspawner scl[9346]: Unknown assignment ExecStartPre=!/usr/bin/chmod -R 0700 /run/jupyter-test-singleuser.
Feb 22 20:09:15 sysdspawner scl[9346]: Failed to create bus message: No such device or address
Feb 22 20:09:16 sysdspawner scl[9346]: [I 2021-02-22 20:09:16.758 JupyterHub log:181] 302 GET /hub/spawn -> /hub/spawn-pending/test (test@::ffff:192.168.56.1) 1008.29ms
Feb 22 20:09:16 sysdspawner scl[9346]: 20:09:16.824 [ConfigProxy] debug: PROXY WEB /hub/spawn-pending/test to http://127.0.0.1:8081
Feb 22 20:09:16 sysdspawner scl[9346]: [I 2021-02-22 20:09:16.832 JupyterHub pages:402] test is pending spawn
Feb 22 20:09:16 sysdspawner scl[9346]: [I 2021-02-22 20:09:16.836 JupyterHub log:181] 200 GET /hub/spawn-pending/test (test@::ffff:192.168.56.1) 8.95ms
Feb 22 20:09:16 sysdspawner scl[9346]: 20:09:16.957 [ConfigProxy] debug: PROXY WEB /hub/api/users/test/server/progress to http://127.0.0.1:8081
Feb 22 20:09:25 sysdspawner scl[9346]: [W 2021-02-22 20:09:25.754 JupyterHub base:1003] User test is slow to start (timeout=10)
Feb 22 20:09:45 sysdspawner scl[9346]: [W 2021-02-22 20:09:45.780 JupyterHub user:681] test's server failed to start in 30 seconds, giving up
Feb 22 20:09:45 sysdspawner scl[9346]: [D 2021-02-22 20:09:45.782 JupyterHub user:790] Stopping test
Feb 22 20:09:45 sysdspawner scl[9346]: [D 2021-02-22 20:09:45.823 JupyterHub user:816] Deleting oauth client jupyterhub-user-test
Feb 22 20:09:45 sysdspawner scl[9346]: [D 2021-02-22 20:09:45.830 JupyterHub user:819] Finished stopping test
Feb 22 20:09:45 sysdspawner scl[9346]: [E 2021-02-22 20:09:45.836 JupyterHub gen:623] Exception in Future <Task finished name='Task-27' coro=<BaseHandler.spawn_single_user.<locals>.fini
Feb 22 20:09:45 sysdspawner scl[9346]: Traceback (most recent call last):
Feb 22 20:09:45 sysdspawner scl[9346]: File "/opt/rh/rh-python38/root/usr/local/lib64/python3.8/site-packages/tornado/gen.py", line 618, in error_callback
Feb 22 20:09:45 sysdspawner scl[9346]: future.result()
Feb 22 20:09:45 sysdspawner scl[9346]: File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/handlers/base.py", line 902, in finish_user_spawn
Feb 22 20:09:45 sysdspawner scl[9346]: await spawn_future
Feb 22 20:09:45 sysdspawner scl[9346]: File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 707, in spawn
Feb 22 20:09:45 sysdspawner scl[9346]: raise e
Feb 22 20:09:45 sysdspawner scl[9346]: File "/opt/rh/rh-python38/root/usr/local/lib/python3.8/site-packages/jupyterhub/user.py", line 606, in spawn
Feb 22 20:09:45 sysdspawner scl[9346]: url = await gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
Feb 22 20:09:45 sysdspawner scl[9346]: tornado.util.TimeoutError: Timeout

I know ExecStartPre is supported in systemd 219 as I've had to use it relatively recently in dealing with an issue on a system.

[behrmann]

I tend to forget, that in the standard setup of systemdspawner jupyterhub runs as root, then one can simply do without the ExecStartPre= and have the hub do the chown.

[bodhar]

Thanks so much for all of the help. It looks like EnvironmentFile pre v228 is only configurable through the unit file, if I'm reading that correctly.

Feb 22 20:58:48 sysdspawner scl[10310]: Unknown assignment EnvironmentFile=/run/jupyter-test-singleuser/jupyter-test-singleuser.env.
Feb 22 20:58:48 sysdspawner scl[10310]: Failed to create bus message: No such device or address

It looks like Environment= may be modified, though (not promising anything there). Would building up Environment= with the various environment file parameters be worth trying?

[behrmann]

Thanks so much for all of the help. It looks like EnvironmentFile pre v228 is only configurable through the unit file, if I'm reading that correctly.

Funny how I remember reading this, that RuntimeDirectory= is not available, but missing the EnvironmentFile=. Then this approach is almost doomed. It could maybe salvaged by setting jupyterhub/services/auth.py:HubAuth the properties that read the config values from environment variables to something that actually reads them from a file. I wanted to try this for an unrelated project anyways, but the turnaround will be a bit longer than yesterday. :)

Generally, it's time to sun set support for systemd 219 and bump it to at least 228.

It looks like Environment= may be modified, though (not promising anything there). Would building up Environment= with the various environment file parameters be worth trying?

Yes, that would work and that is what systemdspawner used to do before and this is just not a good idea, since, quoting from man systemd.exec here,

Note that environment variables are not suitable for passing secrets (such as passwords, key material, ...) to service processes. Environment variables set for a unit are exposed to unprivileged clients via D-Bus IPC, and generally not understood as being data that requires protection. Moreover, environment variables are propagated down the process tree, including across security boundaries (such as setuid/setgid executables), and hence might leak to processes that should not have access to the secret data.

but that's what the whole CVE that 0.15 fixes was about, so it is not advisable.

You might give #40 a go. I haven't updated it in a long time (and it could definitely need that), but it should still work even though it's a bit clunky.

[bodhar]

Thanks for the help! Sorry, some things came up last week.

I'll looking into this, along with the rpm backports, hopefully keeping things at bay till we're able to move away from CentOS 7.

Cheers!