systemdspawner
systemdspawner copied to clipboard
Published
20 hours ago •
jupyterhub
jupyterhub
Use with LDAP authenticator
Bug description
Setup queries LDAP server and launches a JupyterLab instance in the home directory of the user (each user has a Linux user set up already)
Without systemdspawner the setup works perfectly, but as soon as its configured spinning up the user's spawner hangs midway.
I suspect the user needs spawn home directory set in systemdspawner, and their permissions given to the conda environment's jlab executables?
(Corporate Usernames and IP addresses are masked in logs and jhubconfig.py)
Set Up
- OS: RHEL 8 on ppc64le architecture
- Version(s):
jupyter 1.0.0 py39hc1b9086_6 conda-forge
jupyter-lsp 1.4.1 pyhd8ed1ab_0 conda-forge
jupyter-resource-usage 0.6.0 pypi_0 pypi
jupyter-server-mathjax 0.2.3 pypi_0 pypi
jupyter-server-proxy 3.1.0 pypi_0 pypi
jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge
jupyter_console 6.4.0 pyhd8ed1ab_0 conda-forge
jupyter_core 4.7.1 py39hc1b9086_0 conda-forge
jupyter_server 1.10.2 pyhd8ed1ab_0 conda-forge
jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge
jupyterhub 1.4.2 py39hc1b9086_0 conda-forge
jupyterhub-base 1.4.2 py39hc1b9086_0 conda-forge
jupyterhub-idle-culler 1.1 pypi_0 pypi
jupyterhub-ldapauthenticator 1.3.2 py_0 conda-forge
jupyterhub-systemdspawner 0.15.0 pypi_0 pypi
jupyterlab 3.1.6 pyhd8ed1ab_0 conda-forge
jupyterlab-code-formatter 1.4.10 pypi_0 pypi
jupyterlab-drawio 0.9.0 pyhd8ed1ab_0 conda-forge
jupyterlab-git 0.32.0 pypi_0 pypi
jupyterlab-lsp 3.8.1 pyhd8ed1ab_0 conda-forge
jupyterlab-system-monitor 0.8.0 pypi_0 pypi
jupyterlab-topbar 0.6.1 pypi_0 pypi
jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge
jupyterlab_server 2.7.0 pyhd8ed1ab_0 conda-forge
jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge
-
Full environment
_libgcc_mutex 0.1 conda_forge conda-forge _openmp_mutex 4.5 1_gnu conda-forge absl-py 0.13.0 pyhd8ed1ab_0 conda-forge aiohttp 3.7.4.post0 py39ha810350_0 conda-forge alembic 1.6.5 pyhd8ed1ab_0 conda-forge anyio 3.2.1 py39hc1b9086_0 conda-forge appdirs 1.4.4 pypi_0 pypi argon2-cffi 20.1.0 py39ha810350_2 conda-forge astroid 2.6.2 pypi_0 pypi async-timeout 3.0.1 py_1000 conda-forge async_generator 1.10 py_0 conda-forge attrs 21.2.0 pyhd8ed1ab_0 conda-forge autopep8 1.5.7 pyhd8ed1ab_0 conda-forge babel 2.9.1 pyh44b312d_0 conda-forge backcall 0.2.0 pyh9f0ad1d_0 conda-forge backports 1.0 py_2 conda-forge backports.functools_lru_cache 1.6.4 pyhd8ed1ab_0 conda-forge bat 0.18.1 ha57a3a6_0 conda-forge binutils_impl_linux-ppc64le 2.36.1 h5836da8_1 conda-forge black 21.7b0 pypi_0 pypi bleach 3.3.0 pyh44b312d_0 conda-forge blinker 1.4 py_1 conda-forge bokeh 2.3.3 pypi_0 pypi brotlipy 0.7.0 py39ha810350_1001 conda-forge bzip2 1.0.8 h4e0d66e_4 conda-forge c-ares 1.17.1 h4e0d66e_1 conda-forge ca-certificates 2021.5.30 h1084571_0 conda-forge cachetools 4.2.2 pyhd8ed1ab_0 conda-forge certifi 2021.5.30 py39hc1b9086_0 conda-forge certipy 0.1.3 py_0 conda-forge cffi 1.14.6 py39hfce1a3b_0 conda-forge chardet 4.0.0 py39hc1b9086_1 conda-forge click 8.0.1 py39hc1b9086_0 conda-forge cloudpickle 1.6.0 pypi_0 pypi colorama 0.4.4 pyh9f0ad1d_0 conda-forge configurable-http-proxy 4.4.0 node15_h42ff63e_0 conda-forge cryptography 3.4.7 py39h1b86913_0 conda-forge curl 7.78.0 h57b39c4_0 conda-forge dataclasses 0.8 pyhc8e2a94_1 conda-forge debugpy 1.3.0 py39had50986_0 conda-forge decorator 5.0.9 pyhd8ed1ab_0 conda-forge defusedxml 0.7.1 pyhd8ed1ab_0 conda-forge distributed 2021.7.2 pypi_0 pypi docker 5.0.0 pypi_0 pypi dockerspawner 12.1.0 pypi_0 pypi entrypoints 0.3 py39hde42818_1002 conda-forge escapism 1.0.1 pypi_0 pypi exa 0.10.1 ha57a3a6_0 conda-forge expat 2.4.1 h3b9df90_0 conda-forge flake8 3.9.2 pyhd8ed1ab_0 conda-forge freetype 2.10.4 h8a70690_1 conda-forge fsspec 2021.7.0 pypi_0 pypi gcc_impl_linux-ppc64le 9.3.0 ha8fcf76_18 conda-forge gettext 0.19.8.1 hf23c357_1005 conda-forge git 2.32.0 pl5321h29e2a4d_0 conda-forge gitdb 4.0.7 pypi_0 pypi gitpython 3.1.18 pypi_0 pypi google-auth 1.32.1 pyh6c4a22f_0 conda-forge google-auth-oauthlib 0.4.1 py_2 conda-forge greenlet 1.1.0 py39had50986_0 conda-forge grpcio 1.38.1 py39hff9cdcc_0 conda-forge heapdict 1.0.1 pypi_0 pypi icu 68.1 hea85c5d_0 conda-forge idna 2.10 pyh9f0ad1d_0 conda-forge importlib-metadata 4.6.1 py39hc1b9086_0 conda-forge ipykernel 6.0.1 py39h0de2273_0 conda-forge ipython 7.25.0 py39h0de2273_1 conda-forge ipython_genutils 0.2.0 py_1 conda-forge ipywidgets 7.6.3 pyhd3deb0d_0 conda-forge isort 5.9.3 pyhd8ed1ab_0 conda-forge jbig 2.1 h4e0d66e_2003 conda-forge jedi 0.18.0 py39hc1b9086_2 conda-forge jinja2 3.0.1 pyhd8ed1ab_0 conda-forge jpeg 9d h339bb43_0 conda-forge json5 0.9.5 pyh9f0ad1d_0 conda-forge jsonschema 3.2.0 pyhd8ed1ab_3 conda-forge jupyter 1.0.0 py39hc1b9086_6 conda-forge jupyter-lsp 1.4.1 pyhd8ed1ab_0 conda-forge jupyter-resource-usage 0.6.0 pypi_0 pypi jupyter-server-mathjax 0.2.3 pypi_0 pypi jupyter-server-proxy 3.1.0 pypi_0 pypi jupyter_client 6.1.12 pyhd8ed1ab_0 conda-forge jupyter_console 6.4.0 pyhd8ed1ab_0 conda-forge jupyter_core 4.7.1 py39hc1b9086_0 conda-forge jupyter_server 1.10.2 pyhd8ed1ab_0 conda-forge jupyter_telemetry 0.1.0 pyhd8ed1ab_1 conda-forge jupyterhub 1.4.2 py39hc1b9086_0 conda-forge jupyterhub-base 1.4.2 py39hc1b9086_0 conda-forge jupyterhub-idle-culler 1.1 pypi_0 pypi jupyterhub-ldapauthenticator 1.3.2 py_0 conda-forge jupyterhub-systemdspawner 0.15.0 pypi_0 pypi jupyterlab 3.1.6 pyhd8ed1ab_0 conda-forge jupyterlab-code-formatter 1.4.10 pypi_0 pypi jupyterlab-drawio 0.9.0 pyhd8ed1ab_0 conda-forge jupyterlab-git 0.32.0 pypi_0 pypi jupyterlab-lsp 3.8.1 pyhd8ed1ab_0 conda-forge jupyterlab-system-monitor 0.8.0 pypi_0 pypi jupyterlab-topbar 0.6.1 pypi_0 pypi jupyterlab_pygments 0.1.2 pyh9f0ad1d_0 conda-forge jupyterlab_server 2.7.0 pyhd8ed1ab_0 conda-forge jupyterlab_widgets 1.0.0 pyhd8ed1ab_1 conda-forge kernel-headers_linux-ppc64le 3.10.0 hbe0c576_10 conda-forge krb5 1.19.1 haf43566_0 conda-forge lazy-object-proxy 1.6.0 py39ha810350_0 conda-forge lcms2 2.11 h6d9531b_1 conda-forge ld_impl_linux-ppc64le 2.36.1 ha35d02b_1 conda-forge ldap3 2.8.1 pyh9f0ad1d_0 conda-forge lerc 2.2.1 h3b9df90_0 conda-forge libblas 3.9.0 9_openblas conda-forge libcblas 3.9.0 9_openblas conda-forge libcurl 7.78.0 he415e40_0 conda-forge libdeflate 1.7 h4e0d66e_5 conda-forge libedit 3.1.20191231 h41a240f_2 conda-forge libev 4.33 h6eb9509_1 conda-forge libffi 3.3 hea85c5d_2 conda-forge libgcc-devel_linux-ppc64le 9.3.0 h9f60fdd_18 conda-forge libgcc-ng 9.3.0 hc7a5eb4_18 conda-forge libgfortran-ng 9.3.0 hb3da1e1_18 conda-forge libgfortran5 9.3.0 hb3da1e1_18 conda-forge libgomp 9.3.0 hc7a5eb4_18 conda-forge libiconv 1.16 h6eb9509_0 conda-forge libidn2 2.3.1 h4e0d66e_0 conda-forge liblapack 3.9.0 9_openblas conda-forge libnghttp2 1.43.0 h42039ad_0 conda-forge libopenblas 0.3.15 pthreads_h486567c_1 conda-forge libpng 1.6.37 h38e1d09_2 conda-forge libprotobuf 3.17.2 h690f14c_0 conda-forge libsodium 1.0.18 h6eb9509_1 conda-forge libssh2 1.9.0 ha5a9321_6 conda-forge libstdcxx-ng 9.3.0 h22bc1b8_18 conda-forge libtiff 4.3.0 h362c9b0_1 conda-forge libunistring 0.9.10 h14c3975_0 conda-forge libuuid 2.32.1 h4e0d66e_1000 conda-forge libuv 1.41.1 h4e0d66e_0 conda-forge libwebp-base 1.2.0 h4e0d66e_2 conda-forge locket 0.2.1 pypi_0 pypi lz4-c 1.9.3 h3b9df90_0 conda-forge mako 1.1.4 pyh44b312d_0 conda-forge markdown 3.3.4 pyhd8ed1ab_0 conda-forge markupsafe 2.0.1 py39ha810350_0 conda-forge matplotlib-inline 0.1.2 pyhd8ed1ab_2 conda-forge mccabe 0.6.1 pypi_0 pypi mistune 0.8.4 py39ha810350_1004 conda-forge msgpack 1.0.2 pypi_0 pypi multidict 5.1.0 py39ha810350_1 conda-forge mypy-extensions 0.4.3 pypi_0 pypi nbclassic 0.3.1 pyhd8ed1ab_1 conda-forge nbclient 0.5.3 pyhd8ed1ab_0 conda-forge nbconvert 6.1.0 py39hc1b9086_0 conda-forge nbdime 3.1.0 pypi_0 pypi nbformat 5.1.3 pyhd8ed1ab_0 conda-forge ncurses 6.2 hea85c5d_4 conda-forge nest-asyncio 1.5.1 pyhd8ed1ab_0 conda-forge nodejs 15.14.0 h4993e3a_0 conda-forge notebook 6.4.0 pyha770c72_0 conda-forge numpy 1.21.0 py39he089932_0 conda-forge oauthlib 3.1.1 pyhd8ed1ab_0 conda-forge olefile 0.46 pyh9f0ad1d_1 conda-forge openjpeg 2.4.0 h29f4549_1 conda-forge openssl 1.1.1k h4e0d66e_0 conda-forge packaging 21.0 pyhd8ed1ab_0 conda-forge pamela 1.0.0 py_0 conda-forge pandocfilters 1.4.2 py_1 conda-forge parso 0.8.2 pyhd8ed1ab_0 conda-forge partd 1.2.0 pypi_0 pypi pathspec 0.8.1 pypi_0 pypi pcre2 10.37 h32277c3_0 conda-forge perl 5.32.1 0_h4e0d66e_perl5 conda-forge pexpect 4.8.0 pyh9f0ad1d_2 conda-forge pickleshare 0.7.5 py39hde42818_1002 conda-forge pillow 8.3.1 py39hd2e0652_0 conda-forge pip 21.2.4 pyhd8ed1ab_0 conda-forge pluggy 0.13.1 py39hc1b9086_4 conda-forge prometheus_client 0.11.0 pyhd8ed1ab_0 conda-forge prompt-toolkit 3.0.19 pyha770c72_0 conda-forge prompt_toolkit 3.0.19 hd8ed1ab_0 conda-forge protobuf 3.17.2 py39had50986_0 conda-forge psutil 5.8.0 pypi_0 pypi ptyprocess 0.7.0 pyhd3deb0d_0 conda-forge pyasn1 0.4.8 py_0 conda-forge pyasn1-modules 0.2.7 py_0 conda-forge pycodestyle 2.7.0 pyhd8ed1ab_0 conda-forge pycparser 2.20 pyh9f0ad1d_2 conda-forge pycurl 7.43.0.6 py39h2a2158a_1 conda-forge pydocstyle 6.1.1 pyhd8ed1ab_0 conda-forge pyflakes 2.3.1 pyhd8ed1ab_0 conda-forge pygments 2.9.0 pyhd8ed1ab_0 conda-forge pyjwt 2.1.0 pyhd8ed1ab_0 conda-forge pylint 2.9.3 pypi_0 pypi pynvml 11.0.0 pypi_0 pypi pyopenssl 20.0.1 pyhd8ed1ab_0 conda-forge pyparsing 2.4.7 pyh9f0ad1d_0 conda-forge pyrsistent 0.17.3 py39ha810350_2 conda-forge pysocks 1.7.1 py39hc1b9086_3 conda-forge python 3.9.6 h82ac395_0_cpython conda-forge python-dateutil 2.8.1 py_0 conda-forge python-editor 1.0.4 py_0 conda-forge python-json-logger 2.0.1 pyh9f0ad1d_0 conda-forge python-lsp-jsonrpc 1.0.0 pyhd8ed1ab_0 conda-forge python-lsp-server 1.2.1 pypi_0 pypi python_abi 3.9 2_cp39 conda-forge pytz 2021.1 pyhd8ed1ab_0 conda-forge pyu2f 0.1.5 pyhd8ed1ab_0 conda-forge pyyaml 5.4.1 pypi_0 pypi pyzmq 22.1.0 py39h050fd71_0 conda-forge readline 8.1 h5c45dff_0 conda-forge regex 2021.7.6 pypi_0 pypi requests 2.25.1 pyhd3deb0d_0 conda-forge requests-oauthlib 1.3.0 pyh9f0ad1d_0 conda-forge requests-unixsocket 0.2.0 py_0 conda-forge retrolab 0.2.2 pypi_0 pypi rope 0.19.0 pyhd8ed1ab_0 conda-forge rsa 4.7.2 pyh44b312d_0 conda-forge ruamel.yaml 0.17.10 py39ha810350_0 conda-forge ruamel.yaml.clib 0.2.2 py39ha810350_2 conda-forge rust 1.53.0 h978bb50_0 conda-forge rust-std-powerpc64le-unknown-linux-gnu 1.53.0 h6ac4ec8_0 conda-forge send2trash 1.7.1 pyhd8ed1ab_0 conda-forge setuptools 49.6.0 py39hc1b9086_3 conda-forge simpervisor 0.4 pypi_0 pypi six 1.16.0 pyh6c4a22f_0 conda-forge smmap 4.0.0 pypi_0 pypi sniffio 1.2.0 py39hc1b9086_1 conda-forge snowballstemmer 2.1.0 pyhd8ed1ab_0 conda-forge sortedcontainers 2.4.0 pypi_0 pypi sqlalchemy 1.4.20 py39ha810350_0 conda-forge sqlite 3.36.0 h4e2196e_0 conda-forge sysroot_linux-ppc64le 2.17 h8b29623_10 conda-forge tblib 1.7.0 pypi_0 pypi tensorboard 2.4.1 pyhd8ed1ab_0 conda-forge tensorboard-plugin-wit 1.8.0 pyh44b312d_0 conda-forge terminado 0.10.1 py39hc1b9086_0 conda-forge testpath 0.5.0 pyhd8ed1ab_0 conda-forge tk 8.6.10 h38e1d09_1 conda-forge toml 0.10.2 pyhd8ed1ab_0 conda-forge tomli 1.2.1 pypi_0 pypi toolz 0.11.1 pypi_0 pypi tornado 6.1 py39ha810350_1 conda-forge traitlets 5.0.5 py_0 conda-forge typing-extensions 3.10.0.0 hd8ed1ab_0 conda-forge typing_extensions 3.10.0.0 pyha770c72_0 conda-forge tzdata 2021a he74cb21_1 conda-forge ujson 4.0.2 py39had50986_0 conda-forge urllib3 1.26.6 pyhd8ed1ab_0 conda-forge wcwidth 0.2.5 pyh9f0ad1d_2 conda-forge webencodings 0.5.1 py_1 conda-forge websocket-client 0.57.0 py39hc1b9086_4 conda-forge werkzeug 2.0.1 pyhd8ed1ab_0 conda-forge wget 1.20.3 ha5a9321_0 conda-forge wheel 0.36.2 pyhd3deb0d_0 conda-forge widgetsnbextension 3.5.1 py39hc1b9086_4 conda-forge wrapt 1.12.1 py39ha810350_3 conda-forge xeus 1.0.3 h1916c12_1 conda-forge xeus-python 0.12.5 py39h1f10ff4_2 conda-forge xz 5.2.5 h6eb9509_1 conda-forge yapf 0.31.0 pyhd8ed1ab_0 conda-forge yarl 1.6.3 py39ha810350_2 conda-forge zeromq 4.3.4 h3b9df90_0 conda-forge zict 2.0.0 pypi_0 pypi zipp 3.5.0 pyhd8ed1ab_0 conda-forge zlib 1.2.11 h6eb9509_1010 conda-forge zstd 1.5.0 h65c4b1a_0 conda-forge
-
Configuration
import sys c.Spawner.http_timeout = 120 c.Spawner.start_timeout = 120 c.Spawner.cmd=["jupyter-labhub"] c.JupyterHub.authenticator_class = 'ldapauthenticator.LDAPAuthenticator' c.LDAPAuthenticator.server_address = ********** c.LDAPAuthenticator.bind_dn_template = [ "*******\{username}", ] c.LDAPAuthenticator.use_ssl = True c.LDAPAuthenticator.server_port = 636 c.LDAPAuthenticator.valid_username_regex = ".*" c.JupyterHub.services = [ { 'name': 'idle-culler', 'admin': True, 'command': [ sys.executable, '-m', 'jupyterhub_idle_culler', '--timeout=72000' ], } ] c.JupyterHub.admin_access = True c.JupyterHub.bind_url = 'http://:8888' c.JupyterHub.hub_connect_ip = '0.0.0.0' c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
-
Logs
[I 2021-08-14 09:52:37.131 JupyterHub log:189] 302 GET /user/*******/lab -> /hub/user/*******/lab (@********) 0.69ms [W 2021-08-14 09:52:37.348 JupyterHub base:390] Invalid or expired cookie token [I 2021-08-14 09:52:37.349 JupyterHub log:189] 302 GET /hub/user/********/lab -> /hub/login?next=%2Fhub%2Fuser%2F********%2Flab (@********) 0.99ms [I 2021-08-14 09:52:37.568 JupyterHub log:189] 200 GET /hub/login?next=%2Fhub%2Fuser%********%2Flab (@********) 42.46ms [D 2021-08-14 09:52:42.034 JupyterHub ldapauthenticator:379] Attempting to bind ******** with ********\******** [D 2021-08-14 09:52:42.383 JupyterHub ldapauthenticator:392] Status of user bind ******** with ********\******** : True [D 2021-08-14 09:52:42.388 JupyterHub base:530] Setting cookie for ********: jupyterhub-hub-login [D 2021-08-14 09:52:42.388 JupyterHub base:526] Setting cookie jupyterhub-hub-login: {'httponly': True, 'path': '/hub/'} [I 2021-08-14 09:52:42.388 JupyterHub base:762] User logged in: ******** [I 2021-08-14 09:52:42.389 JupyterHub log:189] 302 POST /hub/login?next=%2Fhub%2Fuser%********%2Flab -> /hub/user/********/lab (********@********) 355.25ms [D 2021-08-14 09:52:42.678 JupyterHub user:317] Creating <class 'systemdspawner.systemdspawner.SystemdSpawner'> for ********: [D 2021-08-14 09:52:42.679 JupyterHub systemdspawner:155] user:******** Initialized spawner with unit jupyter-********-singleuser [E 2021-08-14 09:52:42.690 JupyterHub log:189] 503 GET /hub/user/********/lab (********@********) 22.47ms [D 2021-08-14 09:52:42.984 JupyterHub log:189] 200 GET /hub/static/js/not_running.js?v=20210814095231 (@********) 7.36ms [D 2021-08-14 09:52:43.172 JupyterHub log:189] 200 GET /hub/static/js/utils.js?v=20210814095231 (@********) 2.46ms [D 2021-08-14 09:52:45.604 JupyterHub pages:217] Triggering spawn with default options for ******** [D 2021-08-14 09:52:45.604 JupyterHub base:880] Initiating spawn for ******** [D 2021-08-14 09:52:45.604 JupyterHub base:884] 0/100 concurrent spawns [D 2021-08-14 09:52:45.604 JupyterHub base:889] 0 active servers [I 2021-08-14 09:52:45.619 JupyterHub provider:574] Creating oauth client jupyterhub-user-******** [D 2021-08-14 09:52:45.640 JupyterHub user:631] Calling Spawner.start for ******** [D 2021-08-14 09:52:45.640 JupyterHub systemdspawner:201] user:******** Using port 45213 to start spawning user server Running as unit: jupyter-********-singleuser.service [I 2021-08-14 09:52:46.607 JupyterHub log:189] 302 GET /hub/spawn/********?next=%2Fhub%2Fuser%2F********%2Flab -> /hub/spawn-pending/********?next=%2Fhub%2Fuser%2F********%2Flab (********@********) 1005.57ms [I 2021-08-14 09:52:46.864 JupyterHub pages:402] ******** is pending spawn [I 2021-08-14 09:52:46.869 JupyterHub log:189] 200 GET /hub/spawn-pending/********?next=%2Fhub%2Fuser%********%2Flab (********@********) 7.68ms [W 2021-08-14 09:52:55.606 JupyterHub base:1008] User ******** is slow to start (timeout=10)
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
Can you use the PAMAuthenticator instead of the LDAPAuthenticator if the user already has a home directory etc.? I use this setup and that works (for me).
