the-littlest-jupyterhub icon indicating copy to clipboard operation
the-littlest-jupyterhub copied to clipboard

Spawner not running correct python engine TLJH

Open hyamanieu opened this issue 3 years ago • 2 comments

Note: I have set up "the littlest jupyter hub" on Ubuntu 18.04 a few years ago. Everything went well until last week.

Bug description

Using a administrator account, I have update jupyterlab in the base conda. From what I understand, it corresponds to a conda environment under /opt/tljh/user.

Everything looked OK until monday when starting a user server would not "spawn" (time out 30 seconds). I'll copy what I get in journalctl:

Apr 12 09:35:08 jupyterhub-prod systemd[1]: Started /bin/bash -c cd /home/jupyter-hy.amanieu && exec jupyterhub-singleuser --port=58621 --SingleUserNotebookApp.default_url=/lab.
Apr 12 09:35:08 jupyterhub-prod bash[29250]: Traceback (most recent call last):
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/bin/jupyterhub-singleuser", line 5, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from jupyterhub.singleuser import main
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser/__init__.py", line 5, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .app import main
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser/app.py", line 10, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .mixins import make_singleuser_app
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/singleuser/mixins.py", line 45, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from ..log import log_request
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/log.py", line 17, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .handlers.pages import HealthCheckHandler
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/handlers/__init__.py", line 1, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from . import base
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/handlers/base.py", line 32, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .. import orm
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/orm.py", line 42, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from .utils import compare_token
Apr 12 09:35:08 jupyterhub-prod bash[29250]:   File "/opt/tljh/user/lib/python3.7/site-packages/jupyterhub/utils.py", line 24, in <module>
Apr 12 09:35:08 jupyterhub-prod bash[29250]:     from async_generator import aclosing
Apr 12 09:35:08 jupyterhub-prod bash[29250]: ModuleNotFoundError: No module named 'async_generator'
Apr 12 09:35:08 jupyterhub-prod systemd[1]: jupyter-hy.amanieu.service: Main process exited, code=exited, status=1/FAILURE
Apr 12 09:35:08 jupyterhub-prod systemd[1]: jupyter-hy.amanieu.service: Failed with result 'exit-code'.

As you can see, async-generator is missing. Well it is of course not missing from any of my python environment: the hub under /opt/tljh/hub or the main under /opt/tljh/user. I have directly tested the python under the ./bin/ subfolders: all is there.

Typically I would do: <path>/python3 -m pip list | grep async and it would show.

I have checked what is inside the file which starts the spawner: /opt/tljh/user/bin/jupyterhub-singleuser. I could see the following she-bang: #!/opt/tljh/user/bin/python . That engine can import async-generator.

I have no idea how to add more information in the systemd journal for the user to understand which python engine is spawning it and from which packages it's trying to load it.

Expected behaviour

The user server should spawn

Actual behaviour

it doesn't spawn

How to reproduce

I have no idea how to reproduce it. I cannot afford deleting everything to try to reproduce it, this server is important.

Your personal set up

  • OS: 18.04.6 LTS (Bionic Beaver)
  • Version(s): jupyterhub --version : 1.5.0 python: 3.7.5 on the hub ; 3.7.3 on the user's main I used to have python 3.6.9 in the standard path under /usr/bin but I just removed it to try to fix the issue.
Full environment of opt/tljh/user
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(144): Could not remove or rename /opt/tljh/user/conda-meta/widgetsnbextension-3.5.1-pyh9f0ad1d_3.json.  Please remove this file manually (you may need to reboot to free file handles)
WARNING conda.gateways.disk.delete:unlink_or_rename_to_trash(144): Could not remove or rename /opt/tljh/user/conda-meta/ipywidgets-7.6.3-pyhd3deb0d_0.json.  Please remove this file manually (you may need to reboot to free file handles)
# packages in environment at /opt/tljh/user:
#
# Name                    Version                   Build  Channel
_libgcc_mutex             0.1                 conda_forge    conda-forge
_openmp_mutex             4.5                       1_gnu    conda-forge
aiohttp                   3.8.1            py37h540881e_1    conda-forge
aiosignal                 1.2.0              pyhd8ed1ab_0    conda-forge
alembic                   1.4.2                    pypi_0    pypi
anyio                     3.5.0            py37h89c1867_0    conda-forge
argcomplete               2.0.0              pyhd8ed1ab_0    conda-forge
argon2-cffi               21.3.0             pyhd8ed1ab_0    conda-forge
argon2-cffi-bindings      21.2.0           py37h5e8e339_1    conda-forge
async-timeout             4.0.2              pyhd8ed1ab_0    conda-forge
asynctest                 0.13.0                     py_0    conda-forge
attrs                     21.4.0             pyhd8ed1ab_0    conda-forge
babel                     2.9.1              pyh44b312d_0    conda-forge
backcall                  0.2.0              pyh9f0ad1d_0    conda-forge
backports                 1.0                        py_2    conda-forge
backports.functools_lru_cache 1.6.4              pyhd8ed1ab_0    conda-forge
beautifulsoup4            4.11.0             pyha770c72_0    conda-forge
bleach                    3.1.4                    pypi_0    pypi
bokeh                     2.4.2            py37h89c1867_0    conda-forge
bottleneck                1.3.4            py37h6c7ee08_0    conda-forge
bqplot                    0.12.33            pyhd8ed1ab_0    conda-forge
brotli                    1.0.9                h166bdaf_7    conda-forge
brotli-bin                1.0.9                h166bdaf_7    conda-forge
brotli-python             1.0.9            py37hd23a5d3_7    conda-forge
brotlipy                  0.7.0           py37h540881e_1004    conda-forge
bzip2                     1.0.8                h7f98852_4    conda-forge
c-ares                    1.18.1               h7f98852_0    conda-forge
ca-certificates           2022.3.29            h06a4308_0
certifi                   2021.10.8        py37h89c1867_2    conda-forge
certipy                   0.1.3                    pypi_0    pypi
cffi                      1.15.0           py37h7f8727e_0
charset-normalizer        2.0.12             pyhd8ed1ab_0    conda-forge
click                     8.1.2            py37h89c1867_0    conda-forge
colorama                  0.4.4              pyh9f0ad1d_0    conda-forge
conda                     4.10.3           py37h89c1867_4    conda-forge
conda-package-handling    1.8.1            py37h540881e_1    conda-forge
cryptography              36.0.2           py37h38fbfac_1    conda-forge
cycler                    0.11.0             pyhd8ed1ab_0    conda-forge
dash                      2.3.1              pyhd8ed1ab_0    conda-forge
dash-bootstrap-components 1.1.0              pyhd8ed1ab_0    conda-forge
dash-daq                  0.5.0              pyh9f0ad1d_1    conda-forge
dash_colorscales          0.0.4              pyh9f0ad1d_0    conda-forge
dbus                      1.13.6               he372182_0    conda-forge
debugpy                   1.5.1            py37hcd2ae1e_0    conda-forge
decorator                 5.1.1              pyhd8ed1ab_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
dtale                     1.55.0             pyhd8ed1ab_0    conda-forge
entrypoints               0.4                pyhd8ed1ab_0    conda-forge
et_xmlfile                1.0.1                   py_1001    conda-forge
expat                     2.4.8                h27087fc_0    conda-forge
flask                     2.1.1              pyhd8ed1ab_0    conda-forge
flask-compress            1.11               pyhd8ed1ab_0    conda-forge
flit-core                 3.7.1              pyhd8ed1ab_0    conda-forge
fontconfig                2.14.0               h8e229c2_0    conda-forge
fonttools                 4.31.2           py37h540881e_1    conda-forge
freetype                  2.10.4               h0708190_1    conda-forge
frozenlist                1.3.0            py37h540881e_1    conda-forge
future                    0.18.2           py37h89c1867_5    conda-forge
gast                      0.4.0              pyh9f0ad1d_0    conda-forge
gettext                   0.19.8.1          hf34092f_1004    conda-forge
giflib                    5.2.1                h36c2ea0_2    conda-forge
glib                      2.58.3          py37he00f558_1004    conda-forge
gst-plugins-base          1.14.5               h0935bb2_2    conda-forge
gstreamer                 1.14.5               h36ae1b5_2    conda-forge
icu                       58.2              hf484d3e_1000    conda-forge
idna                      3.3                pyhd8ed1ab_0    conda-forge
importlib-metadata        1.6.0                    pypi_0    pypi
importlib_metadata        4.11.3               hd8ed1ab_1    conda-forge
importlib_resources       5.6.0              pyhd8ed1ab_0    conda-forge
ipydatagrid               1.1.2              pyhd8ed1ab_0    conda-forge
ipykernel                 5.2.1                    pypi_0    pypi
ipyparallel               6.3.0            py37h06a4308_0
ipython                   7.13.0                   pypi_0    pypi
ipython_genutils          0.2.0                      py_1    conda-forge
ipywidgets                7.5.1                    pypi_0    pypi
itsdangerous              2.1.2              pyhd8ed1ab_0    conda-forge
jbig                      2.1               h7f98852_2003    conda-forge
jedi                      0.18.1           py37h89c1867_1    conda-forge
jinja2                    3.1.1              pyhd8ed1ab_0    conda-forge
joblib                    1.1.0              pyhd8ed1ab_0    conda-forge
jpeg                      9e                   h7f98852_0    conda-forge
json5                     0.9.4                    pypi_0    pypi
jsonschema                4.4.0              pyhd8ed1ab_0    conda-forge
jupyter-resource-usage    0.6.1                    pypi_0    pypi
jupyter-server-proxy      3.1.0              pyhd8ed1ab_0    conda-forge
jupyter_bokeh             3.0.2              pyhd8ed1ab_0    conda-forge
jupyter_client            7.2.2              pyhd8ed1ab_1    conda-forge
jupyter_core              4.9.2            py37h89c1867_0    conda-forge
jupyter_server            1.13.5             pyhd3eb1b0_0
jupyterhub                1.0.0                    pypi_0    pypi
jupyterlab                1.2.11                   pypi_0    pypi
jupyterlab-server         1.1.1                    pypi_0    pypi
jupyterlab_pygments       0.1.2              pyh9f0ad1d_0    conda-forge
jupyterlab_server         2.12.0             pyhd8ed1ab_0    conda-forge
jupyterlab_widgets        1.1.0              pyhd8ed1ab_0    conda-forge
keyutils                  1.6.1                h166bdaf_0    conda-forge
kiwisolver                1.4.2            py37h7cecad7_1    conda-forge
krb5                      1.19.3               h3790be6_0    conda-forge
lcms2                     2.12                 hddcbb42_0    conda-forge
lerc                      3.0                  h9c3ff4c_0    conda-forge
libarchive                3.5.2                hccf745f_1    conda-forge
libblas                   3.9.0           14_linux64_openblas    conda-forge
libbrotlicommon           1.0.9                h166bdaf_7    conda-forge
libbrotlidec              1.0.9                h166bdaf_7    conda-forge
libbrotlienc              1.0.9                h166bdaf_7    conda-forge
libcblas                  3.9.0           14_linux64_openblas    conda-forge
libcurl                   7.82.0               h7bff187_0    conda-forge
libdeflate                1.10                 h7f98852_0    conda-forge
libedit                   3.1.20191231         he28a2e2_2    conda-forge
libev                     4.33                 h516909a_1    conda-forge
libffi                    3.2.1             he1b5a44_1007    conda-forge
libgcc-ng                 11.2.0              h1d223b6_15    conda-forge
libgfortran-ng            11.2.0              h69a702a_15    conda-forge
libgfortran5              11.2.0              h5c6108e_15    conda-forge
libgomp                   11.2.0              h1d223b6_15    conda-forge
libiconv                  1.16                 h516909a_0    conda-forge
liblapack                 3.9.0           14_linux64_openblas    conda-forge
libnghttp2                1.47.0               h727a467_0    conda-forge
libopenblas               0.3.20          pthreads_h78a6416_0    conda-forge
libpng                    1.6.37               h21135ba_2    conda-forge
libsodium                 1.0.18               h36c2ea0_1    conda-forge
libsolv                   0.7.19               h780b84a_5    conda-forge
libssh2                   1.10.0               ha56f1ee_2    conda-forge
libstdcxx-ng              11.2.0              he4da1e4_15    conda-forge
libtiff                   4.3.0                h542a066_3    conda-forge
libuuid                   2.32.1            h7f98852_1000    conda-forge
libwebp                   1.2.2                h3452ae3_0    conda-forge
libwebp-base              1.2.2                h7f98852_1    conda-forge
libxcb                    1.13              h7f98852_1004    conda-forge
libxml2                   2.9.12               h03d6c58_0
libxslt                   1.1.34               hc22bd24_0
libzlib                   1.2.11            h166bdaf_1014    conda-forge
lxml                      4.6.2            py37h9120a33_0
lz4                       4.0.0            py37hbaff0ff_1    conda-forge
lz4-c                     1.9.3                h9c3ff4c_1    conda-forge
lzo                       2.10              h516909a_1000    conda-forge
mako                      1.1.2                    pypi_0    pypi
mamba                     0.16.0           py37h7f483ca_0    conda-forge
markupsafe                2.1.1            py37h540881e_1    conda-forge
matplotlib                3.5.1            py37h89c1867_0    conda-forge
matplotlib-base           3.5.1            py37h1058ff1_0    conda-forge
matplotlib-inline         0.1.3              pyhd8ed1ab_0    conda-forge
missingno                 0.4.2                      py_1    conda-forge
mistune                   0.8.4           py37h5e8e339_1005    conda-forge
multidict                 6.0.2            py37h540881e_1    conda-forge
munkres                   1.1.4              pyh9f0ad1d_0    conda-forge
nbclassic                 0.3.7              pyhd8ed1ab_0    conda-forge
nbclient                  0.5.13             pyhd8ed1ab_0    conda-forge
nbconvert                 6.4.5              pyhd8ed1ab_2    conda-forge
nbconvert-core            6.4.5              pyhd8ed1ab_2    conda-forge
nbconvert-pandoc          6.4.5              pyhd8ed1ab_2    conda-forge
nbformat                  5.1.3              pyhd8ed1ab_0    conda-forge
nbgitpuller               1.1.0                    pypi_0    pypi
nbresuse                  0.3.4                    pypi_0    pypi
ncurses                   6.3                  h27087fc_1    conda-forge
nest-asyncio              1.5.5              pyhd8ed1ab_0    conda-forge
networkx                  2.7.1              pyhd8ed1ab_0    conda-forge
nodejs                    12.4.0               he1b5a44_0    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
notebook                  6.4.3              pyha770c72_0    conda-forge
notebook-shim             0.1.0              pyhd8ed1ab_0    conda-forge
nteract-on-jupyter        2.1.3                    pypi_0    pypi
numexpr                   2.8.0           py37hfe5f03c_101    conda-forge
numpy                     1.21.5           py37hf2998dd_0    conda-forge
oauthlib                  3.1.0                    pypi_0    pypi
openjpeg                  2.4.0                hb52868f_1    conda-forge
openpyxl                  3.0.9              pyhd8ed1ab_0    conda-forge
openssl                   1.1.1n               h166bdaf_0    conda-forge
packaging                 21.3               pyhd8ed1ab_0    conda-forge
pamela                    1.0.0                    pypi_0    pypi
pandas                    1.3.3            py37h8c16a72_0
pandoc                    2.17.1.1             ha770c72_0    conda-forge
pandocfilters             1.4.2                    pypi_0    pypi
param                     1.12.1             pyh6c4a22f_0    conda-forge
parso                     0.8.3              pyhd8ed1ab_0    conda-forge
patsy                     0.5.2              pyhd8ed1ab_0    conda-forge
pcre                      8.45                 h9c3ff4c_0    conda-forge
pexpect                   4.8.0              pyh9f0ad1d_2    conda-forge
pickleshare               0.7.5                   py_1003    conda-forge
pillow                    9.1.0            py37h44f0d7a_0    conda-forge
pip                       21.2.2           py37h06a4308_0
plotly                    5.7.0              pyhd8ed1ab_0    conda-forge
ppscore                   1.2.0              pyhd3deb0d_0    conda-forge
prometheus-client         0.7.1                    pypi_0    pypi
prometheus_client         0.14.0             pyhd8ed1ab_0    conda-forge
prompt-toolkit            3.0.29             pyha770c72_0    conda-forge
pthread-stubs             0.4               h36c2ea0_1001    conda-forge
ptyprocess                0.7.0              pyhd3deb0d_0    conda-forge
py2vega                   0.6.1              pyhd8ed1ab_0    conda-forge
pycosat                   0.6.3           py37h540881e_1010    conda-forge
pycparser                 2.21               pyhd8ed1ab_0    conda-forge
pygments                  2.11.2             pyhd8ed1ab_0    conda-forge
pyopenssl                 22.0.0             pyhd8ed1ab_0    conda-forge
pyparsing                 3.0.7              pyhd8ed1ab_0    conda-forge
pyqt                      5.9.2            py37hcca6a23_4    conda-forge
pyrsistent                0.18.1           py37h540881e_1    conda-forge
pysocks                   1.7.1            py37h89c1867_5    conda-forge
python                    3.7.3                h0371630_0
python-dateutil           2.8.2              pyhd8ed1ab_0    conda-forge
python-editor             1.0.4                    pypi_0    pypi
python_abi                3.7                     2_cp37m    conda-forge
pytz                      2022.1             pyhd8ed1ab_0    conda-forge
pyviz_comms               2.1.0                      py_0    pyviz
pyyaml                    6.0              py37h540881e_4    conda-forge
pyzmq                     19.0.0                   pypi_0    pypi
qgrid                     1.3.1              pyhd8ed1ab_3    conda-forge
qt                        5.9.7                h52cfd70_2    conda-forge
readline                  7.0               hf8c457e_1001    conda-forge
reproc                    14.2.3               h7f98852_0    conda-forge
reproc-cpp                14.2.3               h9c3ff4c_0    conda-forge
requests                  2.27.1             pyhd8ed1ab_0    conda-forge
rise                      5.7.1            py37h89c1867_0    conda-forge
ruamel_yaml               0.15.80         py37h5e8e339_1006    conda-forge
scikit-learn              1.0.2            py37hf9e9bfc_0    conda-forge
scipy                     1.7.3            py37hf2a6cf1_0    conda-forge
seaborn                   0.11.1             pyhd3eb1b0_0
send2trash                1.8.0              pyhd8ed1ab_0    conda-forge
setuptools                62.0.0           py37h89c1867_0    conda-forge
sharepoint                0.4.2                    pypi_0    pypi
simpervisor               0.4                pyhd8ed1ab_0    conda-forge
sip                       4.19.8           py37hf484d3e_0
six                       1.16.0             pyh6c4a22f_0    conda-forge
sniffio                   1.2.0            py37h89c1867_3    conda-forge
soupsieve                 2.3.1              pyhd8ed1ab_0    conda-forge
sqlalchemy                1.3.16                   pypi_0    pypi
sqlite                    3.33.0               h62c20be_0
squarify                  0.4.3                      py_0    conda-forge
statsmodels               0.13.2           py37hb1e94ed_0    conda-forge
strsimpy                  0.1.9              pyh9f0ad1d_0    conda-forge
tenacity                  8.0.1              pyhd8ed1ab_0    conda-forge
terminado                 0.8.3                    pypi_0    pypi
testpath                  0.6.0              pyhd8ed1ab_0    conda-forge
threadpoolctl             3.1.0              pyh8a188c0_0    conda-forge
tk                        8.6.12               h27826a3_0    conda-forge
tornado                   5.1.1                    pypi_0    pypi
tqdm                      4.64.0             pyhd8ed1ab_0    conda-forge
traitlets                 5.1.1              pyhd8ed1ab_0    conda-forge
traittypes                0.2.1              pyh9f0ad1d_2    conda-forge
typing-extensions         4.1.1                hd8ed1ab_0    conda-forge
typing_extensions         4.1.1              pyha770c72_0    conda-forge
unicodedata2              14.0.0           py37h540881e_1    conda-forge
urllib3                   1.26.9             pyhd8ed1ab_0    conda-forge
wcwidth                   0.1.9                    pypi_0    pypi
webencodings              0.5.1                    pypi_0    pypi
websocket-client          1.3.2              pyhd8ed1ab_0    conda-forge
werkzeug                  2.1.1              pyhd8ed1ab_0    conda-forge
wheel                     0.37.1             pyhd3eb1b0_0
widgetsnbextension        3.6.0                    pypi_0    pypi
xarray                    0.20.2             pyhd8ed1ab_0    conda-forge
xlrd                      2.0.1              pyhd8ed1ab_3    conda-forge
xorg-libxau               1.0.9                h7f98852_0    conda-forge
xorg-libxdmcp             1.1.3                h7f98852_0    conda-forge
xz                        5.2.5                h516909a_1    conda-forge
yaml                      0.2.5                h7f98852_2    conda-forge
yarl                      1.7.2            py37h540881e_2    conda-forge
zeromq                    4.3.4                h9c3ff4c_1    conda-forge
zipp                      3.8.0              pyhd8ed1ab_0    conda-forge
zlib                      1.2.11            h166bdaf_1014    conda-forge
zstd                      1.5.2                ha95c52a_0    conda-forge
/opt/tljh/hub/bin/pip freeze
aiohttp==3.8.1
aiosignal==1.2.0
alembic==1.7.7
anyio==3.5.0
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
async-generator==1.10
async-timeout==4.0.2
asynctest==0.13.0
attrs==21.4.0
Babel==2.9.1
backcall==0.2.0
backoff==1.11.1
bcrypt==3.2.0
beautifulsoup4==4.11.1
bleach==5.0.0
certifi==2021.10.8
certipy==0.1.3
cffi==1.15.0
charset-normalizer==2.0.12
cryptography==36.0.2
debugpy==1.6.0
decorator==5.1.1
deepmerge==1.0.1
defusedxml==0.7.1
entrypoints==0.4
escapism==1.0.1
fastjsonschema==2.15.3
frozenlist==1.3.0
greenlet==1.1.2
idna==3.3
importlib-metadata==4.11.3
importlib-resources==5.6.0
ipykernel==6.13.0
ipython==7.32.0
ipython-genutils==0.2.0
jedi==0.18.1
Jinja2==3.1.1
json5==0.9.6
jsonschema==4.4.0
jupyter-client==7.2.2
jupyter-core==4.9.2
jupyter-server==1.16.0
jupyter-telemetry==0.1.0
jupyterhub==1.5.0
jupyterhub-configurator @ git+https://github.com/yuvipanda/jupyterhub-configurator@317759e17c8e48de1b1352b836dac2a230536dba
jupyterhub-firstuseauthenticator==1.0.0
jupyterhub-idle-culler==1.2.1
jupyterhub-ldapauthenticator==1.3.2
jupyterhub-nativeauthenticator==1.0.5
jupyterhub-systemdspawner==0.15.0
jupyterhub-tmpauthenticator==0.6
jupyterhub-traefik-proxy==0.3.0
jupyterlab==3.3.3
jupyterlab-pygments==0.2.0
jupyterlab-server==2.12.0
ldap3==2.9.1
Mako==1.2.0
MarkupSafe==2.1.1
matplotlib-inline==0.1.3
mistune==0.8.4
multidict==6.0.2
nbclassic==0.3.7
nbclient==0.5.13
nbconvert==6.5.0
nbformat==5.3.0
nest-asyncio==1.5.5
notebook==6.4.10
notebook-shim==0.1.0
oauthenticator==14.2.0
oauthlib==3.2.0
onetimepass==1.0.1
packaging==21.3
pamela==1.0.0
pandocfilters==1.5.0
parso==0.8.3
passlib==1.7.4
pexpect==4.8.0
pickleshare==0.7.5
pluggy==1.0.0
prometheus-client==0.14.1
prompt-toolkit==3.0.29
psutil==5.9.0
ptyprocess==0.7.0
pyasn1==0.4.8
pycparser==2.21
pycurl==7.45.1
Pygments==2.11.2
pyOpenSSL==22.0.0
pyparsing==3.0.8
pyrsistent==0.18.1
python-dateutil==2.8.2
python-json-logger==2.0.2
pytz==2022.1
pyzmq==22.3.0
requests==2.27.1
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
Send2Trash==1.8.0
six==1.16.0
sniffio==1.2.0
soupsieve==2.3.2
SQLAlchemy==1.4.35
terminado==0.13.3
the-littlest-jupyterhub @ git+https://github.com/jupyterhub/the-littlest-jupyterhub.git@0572976d25c18bc88d936f8bd8627bbfed069b53
tinycss2==1.1.1
toml==0.10.2
tornado==6.1
traitlets==5.1.1
typing_extensions==4.1.1
urllib3==1.26.9
wcwidth==0.2.5
webencodings==0.5.1
websocket-client==1.3.2
yarl==1.7.2
zipp==3.8.0

hyamanieu avatar Apr 12 '22 10:04 hyamanieu

As this is an Issue which normally would require a PR to correct, I'll refine my needs:

  • Would it be possible to add documentation about adding the python engine path or cwd in the user service journal?
  • Would it be possible to add documentation on how this engine is selected by the user service spawner?

Thank you

hyamanieu avatar Apr 19 '22 07:04 hyamanieu

I think I am seeing a similar issue. I'll admit upfront that I am new to JuptyerHub, and by extension TLJH.

I was trying to add the bits for AzureAD authentication to the config. I kept getting exceptions on startup about not being able to import jwt. But when I was in the venv, pip freeze | grep -i jwt revealed that both jwt and PyJWT were installed. Running the same pip from outside of the venv only showed PyJWT installed.

This lead me to suspect that oauthenticator, or at least the AzureADAuthenticator class is using the system python rather than the python from within the virtual env.

When did pip3 install jwt as root from outside of the venv, I stopped getting the exception.

But JH still doesn't start and the logging isn't providing anything helpful. I assume that it is more failures from packages due to it using the system python.

I am toying with the idea of snapshotting the VM, and then linking the system python binaries directly into the venv to see if anything changes.

I'm unsure how to proceed, aside from scraping the SSO component.

kgleason avatar Oct 14 '22 14:10 kgleason