jupyterhub
jupyterhub copied to clipboard
Published
20 hours ago •
jupyterhub
jupyterhub
Persistent 503 errors for some users, but not all
Bug description
I have a jupyterhub server up and running, with users able to log on, as am I. However some users get a 503 error and are unable to log on, and it is not temporary - the problem is persistent.
If I remove the user account, and get them to log on again and recreate the account the problem goes away.
Expected behaviour
The users should be able to log on
Actual behaviour
The user gets a 503 error
How to reproduce
Your personal set up
Installed using conda with users authenticating using Azure AAD
conda install -c conda-forge jupyterhub -y
conda install notebook -y
# Install oauth etc for aad auth
conda install -c conda-forge oauthenticator pyJWT=1.7.1 jupyterhub-ldapauthenticator -y
# Also require systemdspawner
conda install -c conda-forge jupyterhub-systemdspawner -y
# Add idle user culler script
conda install -c conda-forge jupyterhub-idle-culler -y
# Add nbextensions for e.g. spellcheck and SKILL syntax
conda install -c conda-forge jupyter_contrib_nbextensions -y
- OS: NAME="Ubuntu" VERSION="18.04.5 LTS (Bionic Beaver)"
- Version(s):
jupyterhub 1.4.2 Python 3.8.12
Full environment
# packages in environment at /opt/anaconda3:
#
# Name Version Build Channel
_anaconda_depends 2021.11 py38_0
_ipyw_jlab_nb_ext_conf 0.1.0 py38_0
_libgcc_mutex 0.1 main
_openmp_mutex 4.5 1_gnu
alabaster 0.7.12 pyhd3eb1b0_0
alembic 1.7.5 pyhd3eb1b0_1
anaconda custom py38_1
anaconda-client 1.9.0 py38h06a4308_0
anaconda-navigator 2.0.4 py38_0
anaconda-project 0.10.2 pyhd3eb1b0_0
anyio 2.2.0 py38h06a4308_1
appdirs 1.4.4 pyhd3eb1b0_0
argh 0.26.2 py38_0
argon2-cffi 20.1.0 py38h27cfd23_1
arrow 0.13.1 py38_0
asn1crypto 1.4.0 py_0
astroid 2.6.6 py38h06a4308_0
astropy 4.3.1 py38h09021b7_0
async_generator 1.10 pyhd3eb1b0_0
atomicwrites 1.4.0 py_0
attrs 21.2.0 pyhd3eb1b0_0
autopep8 1.5.7 pyhd3eb1b0_0
babel 2.9.1 pyhd3eb1b0_0
backcall 0.2.0 pyhd3eb1b0_0
backports 1.0 pyhd3eb1b0_2
backports.functools_lru_cache 1.6.4 pyhd3eb1b0_0
backports.shutil_get_terminal_size 1.0.0 pyhd3eb1b0_3
backports.tempfile 1.0 pyhd3eb1b0_1
backports.weakref 1.0.post1 py_1
beautifulsoup4 4.10.0 pyh06a4308_0
binaryornot 0.4.4 pyhd3eb1b0_1
bitarray 2.3.0 py38h7f8727e_1
bkcharts 0.2 py38_0
black 19.10b0 py_0
blas 1.0 mkl
bleach 4.0.0 pyhd3eb1b0_0
blinker 1.4 py38h06a4308_0
blosc 1.21.0 h8c45485_0
bokeh 2.4.2 py38h06a4308_0
boto 2.49.0 py38_0
bottleneck 1.3.2 py38heb32a55_1
branca 0.4.2 pypi_0 pypi
brotli 1.0.9 he6710b0_2
brotlipy 0.7.0 py38h27cfd23_1003
brunsli 0.1 h2531618_0
bzip2 1.0.8 h7b6447c_0
c-ares 1.17.1 h27cfd23_0
ca-certificates 2021.10.26 h06a4308_2
cairo 1.16.0 hf32fb01_1
certifi 2021.10.8 py38h06a4308_0
certipy 0.1.3 pyhd3eb1b0_0
cffi 1.14.6 py38h400218f_0
cfitsio 3.470 hf0d0db6_6
chardet 4.0.0 py38h06a4308_1003
charls 2.2.0 h2531618_0
charset-normalizer 2.0.4 pyhd3eb1b0_0
click 8.0.3 pyhd3eb1b0_0
click-plugins 1.1.1 pypi_0 pypi
cligj 0.7.2 pypi_0 pypi
cloudpickle 2.0.0 pyhd3eb1b0_0
clyent 1.2.2 py38_1
colorama 0.4.4 pyhd3eb1b0_0
conda 4.11.0 py38h578d9bd_0 conda-forge
conda-build 3.21.7 py38h06a4308_0
conda-content-trust 0.1.1 pyhd3eb1b0_0
conda-env 2.6.0 1
conda-pack 0.6.0 pyhd3eb1b0_0
conda-package-handling 1.7.3 py38h27cfd23_1
conda-repo-cli 1.0.4 pyhd3eb1b0_0
conda-token 0.3.0 pyhd3eb1b0_0
conda-verify 3.4.2 py_1
configurable-http-proxy 4.0.1 node6_0
contextlib2 0.6.0.post1 pyhd3eb1b0_0
cookiecutter 1.7.2 pyhd3eb1b0_0
cryptography 3.4.8 py38hd23ed53_0
curl 7.78.0 h1ccaba5_0
cycler 0.11.0 pyhd3eb1b0_0
cython 0.29.24 py38hdbfa776_0
cytoolz 0.11.0 py38h7b6447c_0
daal4py 2021.4.0 py38h78b71dc_0
dal 2021.4.0 h06a4308_729
dask 2021.10.0 pyhd3eb1b0_0
dask-core 2021.10.0 pyhd3eb1b0_0
dataclasses 0.8 pyh6d0b6a4_7
dbus 1.13.18 hb2f20db_0
debugpy 1.5.1 py38h295c915_0
decorator 5.1.0 pyhd3eb1b0_0
defusedxml 0.7.1 pyhd3eb1b0_0
diff-match-patch 20200713 pyhd3eb1b0_0
distributed 2021.10.0 py38h06a4308_0
docutils 0.17.1 py38h06a4308_1
entrypoints 0.3 py38_0
et_xmlfile 1.1.0 py38h06a4308_0
expat 2.4.1 h2531618_2
fastcache 1.1.0 py38h7b6447c_0
filelock 3.4.0 pyhd3eb1b0_0
fiona 1.8.20 pypi_0 pypi
flake8 3.9.2 pyhd3eb1b0_0
flask 1.1.2 pyhd3eb1b0_0
folium 0.12.1 pypi_0 pypi
fontconfig 2.13.1 hba837de_1005 conda-forge
fonttools 4.25.0 pyhd3eb1b0_0
freetype 2.11.0 h70c0345_0
fribidi 1.0.10 h7b6447c_0
fsspec 2021.10.1 pyhd3eb1b0_0
future 0.18.2 py38_1
geopandas 0.10.2 pypi_0 pypi
get_terminal_size 1.0.0 haa9412d_0
gevent 21.8.0 py38h7f8727e_1
giflib 5.2.1 h7b6447c_0
glib 2.69.1 h5202010_0
glob2 0.7 pyhd3eb1b0_0
gmp 6.2.1 h2531618_2
gmpy2 2.0.8 py38hd5f6e3b_3
graphite2 1.3.14 h23475e2_0
greenlet 1.1.1 py38h295c915_0
gst-plugins-base 1.14.0 h8213a91_2
gstreamer 1.14.0 h28cd5cc_2
h5py 2.10.0 py38h7918eee_0
harfbuzz 2.8.1 h6f93f22_0
hdf5 1.10.4 hb1b8bf9_0
heapdict 1.0.1 pyhd3eb1b0_0
html5lib 1.1 pyhd3eb1b0_0
icu 58.2 he6710b0_3
idna 3.3 pyhd3eb1b0_0
imagecodecs 2021.8.26 py38h4cda21f_0
imageio 2.9.0 pyhd3eb1b0_0
imagesize 1.3.0 pyhd3eb1b0_0
importlib-metadata 4.8.2 py38h06a4308_0
importlib_metadata 4.8.2 hd3eb1b0_0
importlib_resources 5.2.0 pyhd3eb1b0_1
inflection 0.5.1 py38h06a4308_0
iniconfig 1.1.1 pyhd3eb1b0_0
intel-openmp 2021.4.0 h06a4308_3561
intervaltree 3.1.0 pyhd3eb1b0_0
ipykernel 6.4.1 py38h06a4308_1
ipython 7.29.0 py38hb070fc8_0
ipython_genutils 0.2.0 pyhd3eb1b0_1
ipywidgets 7.6.5 pyhd3eb1b0_1
isort 5.9.3 pyhd3eb1b0_0
itsdangerous 2.0.1 pyhd3eb1b0_0
jbig 2.1 hdba287a_0
jdcal 1.4.1 pyhd3eb1b0_0
jedi 0.18.0 py38h06a4308_1
jeepney 0.7.1 pyhd3eb1b0_0
jinja2 2.11.3 pyhd3eb1b0_0
jinja2-time 0.2.0 pyhd3eb1b0_2
joblib 1.1.0 pyhd3eb1b0_0
jpeg 9d h7f8727e_0
json5 0.9.6 pyhd3eb1b0_0
jsonschema 3.2.0 pyhd3eb1b0_2
jupyter 1.0.0 py38_7
jupyter-pytest-2 1.0.1 pypi_0 pypi
jupyter_client 6.1.12 pyhd3eb1b0_0
jupyter_console 6.4.0 pyhd3eb1b0_0
jupyter_contrib_core 0.3.3 py_2 conda-forge
jupyter_contrib_nbextensions 0.5.1 pyhd8ed1ab_2 conda-forge
jupyter_core 4.9.1 py38h06a4308_0
jupyter_highlight_selected_word 0.2.0 py38h578d9bd_1002 conda-forge
jupyter_latex_envs 1.4.6 pyhd8ed1ab_1002 conda-forge
jupyter_nbextensions_configurator 0.4.1 py38h578d9bd_2 conda-forge
jupyter_server 1.4.1 py38h06a4308_0
jupyter_telemetry 0.1.0 py_0
jupyterhub 1.4.2 py38h06a4308_0
jupyterhub-idle-culler 1.0 pyh9f0ad1d_0 conda-forge
jupyterhub-ldapauthenticator 1.3.2 pyhd3eb1b0_0
jupyterhub-systemdspawner 0.13 py38h32f6830_1 conda-forge
jupyterlab 3.2.1 pyhd3eb1b0_1
jupyterlab_pygments 0.1.2 py_0
jupyterlab_server 2.8.2 pyhd3eb1b0_0
jupyterlab_widgets 1.0.0 pyhd3eb1b0_1
jxrlib 1.1 h7b6447c_2
keyring 23.4.0 py38h06a4308_0
kiwisolver 1.3.1 py38h2531618_0
krb5 1.19.2 hac12032_0
lazy-object-proxy 1.6.0 py38h27cfd23_0
lcms2 2.12 h3be6417_0
ld_impl_linux-64 2.35.1 h7274673_9
ldap3 2.9.1 pyhd3eb1b0_0
lerc 3.0 h295c915_0
libaec 1.0.4 he6710b0_1
libarchive 3.4.2 h62408e4_0
libcurl 7.78.0 h0b77cf5_0
libdeflate 1.8 h7f8727e_5
libedit 3.1.20210910 h7f8727e_0
libev 4.33 h7f8727e_1
libffi 3.3 he6710b0_2
libgcc 7.2.0 h69d50b8_2
libgcc-ng 9.3.0 h5101ec6_17
libgfortran-ng 7.5.0 ha8ba4b0_17
libgfortran4 7.5.0 ha8ba4b0_17
libgomp 9.3.0 h5101ec6_17
libiconv 1.15 h63c8f33_5
liblief 0.10.1 he6710b0_0
libllvm11 11.1.0 h3826bc1_0
libllvm9 9.0.1 h4a3c616_1
libnghttp2 1.46.0 hce63b2e_0
libpng 1.6.37 hbc83047_0
libsodium 1.0.18 h7b6447c_0
libsolv 0.7.16 h8b12597_0 conda-forge
libspatialindex 1.9.3 h2531618_0
libssh2 1.9.0 h1ba5d50_1
libstdcxx-ng 9.3.0 hd4cf53a_17
libtiff 4.2.0 h85742a9_0
libtool 2.4.6 h7b6447c_1005
libuuid 2.32.1 h14c3975_1000 conda-forge
libuv 1.40.0 h7b6447c_0
libwebp 1.2.0 h89dd481_0
libwebp-base 1.2.0 h27cfd23_0
libxcb 1.14 h7b6447c_0
libxml2 2.9.12 h03d6c58_0
libxslt 1.1.34 hc22bd24_0
libzopfli 1.0.3 he6710b0_0
llvmlite 0.33.0 py38hc6ec683_1
locket 0.2.1 py38h06a4308_1
lxml 4.6.3 py38h9120a33_0
lz4-c 1.9.3 h295c915_1
lzo 2.10 h7b6447c_2
mako 1.1.4 pyhd3eb1b0_0
mamba 0.5.1 py38h6fd9b40_0 conda-forge
markupsafe 1.1.1 py38h7b6447c_0
matplotlib 3.5.0 py38h06a4308_0
matplotlib-base 3.5.0 py38h3ed280b_0
matplotlib-inline 0.1.2 pyhd3eb1b0_2
mccabe 0.6.1 py38_1
mistune 0.8.4 py38h7b6447c_1000
mkl 2021.4.0 h06a4308_640
mkl-service 2.4.0 py38h7f8727e_0
mkl_fft 1.3.1 py38hd3c417c_0
mkl_random 1.2.2 py38h51133e4_0
mock 4.0.3 pyhd3eb1b0_0
more-itertools 8.12.0 pyhd3eb1b0_0
mpc 1.1.0 h10f8cd9_1
mpfr 4.0.2 hb69a4c5_1
mpi 1.0 mpich
mpich 3.3.2 hc856adb_0
mpmath 1.2.1 py38h06a4308_0
msgpack-python 1.0.2 py38hff7bd54_1
multipledispatch 0.6.0 py38_0
munch 2.5.0 pypi_0 pypi
munkres 1.1.4 py_0
mypy_extensions 0.4.3 py38_0
navigator-updater 0.2.1 py38_0
nb_conda_kernels 2.3.1 py38h06a4308_0
nbclassic 0.2.6 pyhd3eb1b0_0
nbclient 0.5.3 pyhd3eb1b0_0
nbconvert 6.1.0 py38h06a4308_0
nbformat 5.1.3 pyhd3eb1b0_0
ncurses 6.3 h7f8727e_2
nest-asyncio 1.5.1 pyhd3eb1b0_0
networkx 2.6.3 pyhd3eb1b0_0
nltk 3.6.5 pyhd3eb1b0_0
nodejs 6.11.2 h3db8ef7_0
nose 1.3.7 pyhd3eb1b0_1006
notebook 6.4.6 py38h06a4308_0
numba 0.50.1 py38h0573a6f_1
numexpr 2.7.3 py38h22e1b3c_1
numpy 1.21.2 py38h20f2e39_0
numpy-base 1.21.2 py38h79a1101_0
numpydoc 1.1.0 pyhd3eb1b0_1
oauthenticator 14.0.0 pyhd8ed1ab_0 conda-forge
oauthlib 3.1.0 py_0
olefile 0.46 pyhd3eb1b0_0
openjpeg 2.4.0 h3ad879b_0
openpyxl 3.0.9 pyhd3eb1b0_0
openssl 1.1.1l h7f8727e_0
ordo 0.3.4 pypi_0 pypi
packaging 21.3 pyhd3eb1b0_0
pamela 1.0.0 pyhd3eb1b0_0
pandas 1.3.4 py38h8c16a72_0
pandocfilters 1.4.3 py38h06a4308_1
pango 1.45.3 hd140c19_0
parso 0.8.2 pyhd3eb1b0_0
partd 1.2.0 pyhd3eb1b0_0
patchelf 0.13 h295c915_0
path 16.0.0 py38h06a4308_0
path.py 12.5.0 hd3eb1b0_0
pathlib2 2.3.6 py38h06a4308_2
pathspec 0.7.0 py_0
patsy 0.5.2 py38h06a4308_0
pcre 8.45 h295c915_0
pep8 1.7.1 py38_0
pexpect 4.8.0 pyhd3eb1b0_3
pickleshare 0.7.5 pyhd3eb1b0_1003
pillow 8.4.0 py38h5aabda8_0
pip 21.2.4 py38h06a4308_0
pixman 0.40.0 h7f8727e_1
pkginfo 1.7.1 py38h06a4308_0
pluggy 1.0.0 py38h06a4308_0
ply 3.11 py38_0
poyo 0.5.0 pyhd3eb1b0_0
prometheus_client 0.12.0 pyhd3eb1b0_0
prompt-toolkit 3.0.20 pyhd3eb1b0_0
prompt_toolkit 3.0.20 hd3eb1b0_0
psutil 5.8.0 py38h27cfd23_1
ptyprocess 0.7.0 pyhd3eb1b0_2
py 1.10.0 pyhd3eb1b0_0
py-lief 0.10.1 py38h403a769_0
pyasn1 0.4.8 pyhd3eb1b0_0
pycodestyle 2.7.0 pyhd3eb1b0_0
pycosat 0.6.3 py38h7b6447c_1
pycparser 2.21 pyhd3eb1b0_0
pycurl 7.44.1 py38h8f2d780_1
pydocstyle 6.1.1 pyhd3eb1b0_0
pyerfa 2.0.0 py38h27cfd23_0
pyflakes 2.3.1 pyhd3eb1b0_0
pygments 2.10.0 pyhd3eb1b0_0
pyjwt 1.7.1 py38_0
pylint 2.9.6 py38h06a4308_1
pyls-spyder 0.4.0 pyhd3eb1b0_0
pyodbc 4.0.31 py38h295c915_0
pyopenssl 21.0.0 pyhd3eb1b0_1
pyparsing 3.0.4 pyhd3eb1b0_0
pyproj 3.3.0 pypi_0 pypi
pyqt 5.9.2 py38h05f1152_4
pyrsistent 0.18.0 py38heee7806_0
pysocks 1.7.1 py38h06a4308_0
pytables 3.6.1 py38h9fd0a39_0
pytest 6.2.5 py38h06a4308_2
pytest-timeout 1.4.2 pypi_0 pypi
python 3.8.12 h12debd9_0
python-dateutil 2.8.2 pyhd3eb1b0_0
python-json-logger 2.0.1 py_0
python-libarchive-c 2.9 pyhd3eb1b0_1
python-lsp-black 1.0.0 pyhd3eb1b0_0
python-lsp-jsonrpc 1.0.0 pyhd3eb1b0_0
python-lsp-server 1.2.4 pyhd3eb1b0_0
python-slugify 5.0.2 pyhd3eb1b0_0
python_abi 3.8 1_cp38 conda-forge
pytz 2021.3 pyhd3eb1b0_0
pywavelets 1.1.1 py38h7b6447c_2
pyxdg 0.27 pyhd3eb1b0_0
pyyaml 6.0 py38h7f8727e_1
pyzmq 22.3.0 py38h295c915_2
qdarkstyle 3.0.2 pyhd3eb1b0_0
qstylizer 0.1.10 pyhd3eb1b0_0
qt 5.9.7 h5867ecd_1
qtawesome 1.0.3 pyhd3eb1b0_0
qtconsole 5.1.1 pyhd3eb1b0_0
qtpy 1.10.0 pyhd3eb1b0_0
readline 8.1 h27cfd23_0
regex 2021.8.3 py38h7f8727e_0
requests 2.26.0 pyhd3eb1b0_0
ripgrep 12.1.1 0
rope 0.21.1 pyhd3eb1b0_0
rtree 0.9.7 py38h06a4308_1
ruamel.yaml 0.16.12 py38h7b6447c_1
ruamel.yaml.clib 0.2.2 py38h7b6447c_0
ruamel_yaml 0.15.100 py38h27cfd23_0
scikit-image 0.18.3 py38h51133e4_0
scikit-learn 1.0.1 py38h51133e4_0
scikit-learn-intelex 2021.4.0 py38h06a4308_0
scipy 1.7.1 py38h292c36d_2
seaborn 0.11.2 pyhd3eb1b0_0
secretstorage 3.3.1 py38h06a4308_0
send2trash 1.8.0 pyhd3eb1b0_1
setuptools 58.0.4 py38h06a4308_0
shapely 1.8.0 pypi_0 pypi
simplegeneric 0.8.1 py38_2
singledispatch 3.7.0 pyhd3eb1b0_1001
sip 4.19.13 py38he6710b0_0
six 1.16.0 pyhd3eb1b0_0
snappy 1.1.8 he6710b0_0
sniffio 1.2.0 py38h06a4308_1
snowballstemmer 2.2.0 pyhd3eb1b0_0
sortedcollections 2.1.0 pyhd3eb1b0_0
sortedcontainers 2.4.0 pyhd3eb1b0_0
soupsieve 2.3.1 pyhd3eb1b0_0
sphinx 4.2.0 pyhd3eb1b0_1
sphinxcontrib 1.0 py38_1
sphinxcontrib-applehelp 1.0.2 pyhd3eb1b0_0
sphinxcontrib-devhelp 1.0.2 pyhd3eb1b0_0
sphinxcontrib-htmlhelp 2.0.0 pyhd3eb1b0_0
sphinxcontrib-jsmath 1.0.1 pyhd3eb1b0_0
sphinxcontrib-qthelp 1.0.3 pyhd3eb1b0_0
sphinxcontrib-serializinghtml 1.1.5 pyhd3eb1b0_0
sphinxcontrib-websupport 1.2.4 py_0
spyder 5.1.5 py38h06a4308_1
spyder-kernels 2.1.3 py38h06a4308_0
sqlalchemy 1.4.27 py38h7f8727e_0
sqlite 3.36.0 hc218d9a_0
statsmodels 0.13.0 py38h7f8727e_0
sympy 1.9 py38h06a4308_0
tbb 2021.4.0 hd09550d_0
tbb4py 2021.4.0 py38hd09550d_0
tblib 1.7.0 pyhd3eb1b0_0
terminado 0.9.4 py38h06a4308_0
testpath 0.5.0 pyhd3eb1b0_0
text-unidecode 1.3 pyhd3eb1b0_0
textdistance 4.2.1 pyhd3eb1b0_0
threadpoolctl 2.2.0 pyh0d69192_0
three-merge 0.1.1 pyhd3eb1b0_0
tifffile 2021.7.2 pyhd3eb1b0_2
tinycss 0.4 pyhd3eb1b0_1002
tk 8.6.11 h1ccaba5_0
toml 0.10.2 pyhd3eb1b0_0
toolz 0.11.2 pyhd3eb1b0_0
tornado 6.1 py38h27cfd23_0
tqdm 4.62.3 pyhd3eb1b0_1
traitlets 5.1.1 pyhd3eb1b0_0
typed-ast 1.4.3 py38h7f8727e_1
typing_extensions 3.10.0.2 pyh06a4308_0
ujson 4.0.2 py38h2531618_0
unicodecsv 0.14.1 py38_0
unidecode 1.2.0 pyhd3eb1b0_0
unixodbc 2.3.9 h7b6447c_0
urllib3 1.26.7 pyhd3eb1b0_0
watchdog 2.1.6 py38h06a4308_0
wcwidth 0.2.5 pyhd3eb1b0_0
webencodings 0.5.1 py38_1
werkzeug 2.0.2 pyhd3eb1b0_0
wheel 0.37.0 pyhd3eb1b0_1
whichcraft 0.6.1 pyhd3eb1b0_0
widgetsnbextension 3.5.1 py38_0
wrapt 1.12.1 py38h7b6447c_1
wurlitzer 3.0.2 py38h06a4308_0
xeus 1.0.1 h7d0c39e_2 conda-forge
xeus-python 0.12.3 py38h56bed4f_0 conda-forge
xlrd 2.0.1 pyhd3eb1b0_0
xlsxwriter 3.0.2 pyhd3eb1b0_0
xlwt 1.3.0 py38_0
xmltodict 0.12.0 pyhd3eb1b0_0
xz 5.2.5 h7b6447c_0
yaml 0.2.5 h7b6447c_0
yapf 0.31.0 pyhd3eb1b0_0
zeromq 4.3.4 h2531618_0
zfp 0.5.5 h2531618_6
zict 2.0.0 pyhd3eb1b0_0
zipp 3.6.0 pyhd3eb1b0_0
zlib 1.2.11 h7b6447c_3
zope 1.0 py38_1
zope.event 4.5.0 py38_0
zope.interface 5.4.0 py38h7f8727e_0
zstd 1.4.9 haebb681_0
Configuration
import os
import sys
from oauthenticator.azuread import LocalAzureAdOAuthenticator
from oauthenticator import LocalAuthenticator
import systemdspawner
# Debug level
c.Application.log_level = 'DEBUG'
# Network & security
c.JupyterHub.port = 443
c.JupyterHub.ssl_key = '/etc/letsencrypt/live/addresshere/privkey.pem'
c.JupyterHub.ssl_cert = '/etc/letsencrypt/live/addresshere/fullchain.pem'
c.ConfigurableHTTPProxy.command = ['configurable-http-proxy', '--redirect-port', '80']
# Spawning
c.JupyterHub.spawner_class = 'systemdspawner.SystemdSpawner'
c.SystemdSpawner.user_workingdir = '/jupyter/{USERNAME}'
c.SystemdSpawner.username_template = '{USERNAME}'
c.SystemdSpawner.cmd=["jupyter-labhub"]
# Cull idle users (new section)
c.JupyterHub.services = [
{
'name': 'idle-culler',
'admin': True,
'command': [
sys.executable,
'-m', 'jupyterhub_idle_culler',
'--timeout=3600'
],
}
]
c.SystemdSpawner.start_timeout = 400
c.SystemdSpawner.http_timeout = 400
c.SystemdSpawner.poll_interval = 400
c.JupyterHub.tornado_settings = {
'slow_spawn_timeout': 10
}
c.SystemdSpawner.mem_limit = '0.5G'
c.SystemdSpawner.cpu_limit = 0.45
# Authenication
c.JupyterHub.authenticator_class = LocalAzureAdOAuthenticator
c.LocalAzureAdOAuthenticator.tenant_id = os.environ.get('AAD_TENANT_ID')
c.LocalAzureAdOAuthenticator.oauth_callback_url = os.environ.get('AAD_CALLBACK')
c.LocalAzureAdOAuthenticator.client_id = os.environ.get('AAD_APP_CLIENT_ID')
c.LocalAzureAdOAuthenticator.client_secret = os.environ.get('AAD_APP_CLIENT_SECRET')
c.LocalAzureAdOAuthenticator.username_claim = 'unique_name'
Logs
I get lots of 503 errors in the logs like this:
Apr 4 14:09:21 res-dev-jh-maths-vm bash[3774]: [I 2022-04-04 14:09:21.273 SingleUserLabApp kernelmanager:187] Kernel started: e8724899-ce29-4d1d-a553-ade835c5c6c0
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.273 JupyterHub pages:652] No template for 503
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.274 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fkernels%2F022770c5-6b15-49a5-a080-72ec66cf9beb%2Fchannels%3Fsession_id%3De7d76c93-fd50-4e68-acd9-d5b979174393 (@127.0.0.1) 2.01ms
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: 14:09:21.344 [ConfigProxy] #033[31merror#033[39m: 503 GET /user/jupyter-thh64/api/kernels/d124afa2-663c-4d6d-ad1b-6d7f3fa779c3/channels?session_id=fc2a6e85-d8aa-4625-8e59-67324186f2cb connect ECONNREFUSED 127.0.0.1:56899
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.345 JupyterHub pages:652] No template for 503
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.347 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fkernels%2Fd124afa2-663c-4d6d-ad1b-6d7f3fa779c3%2Fchannels%3Fsession_id%3Dfc2a6e85-d8aa-4625-8e59-67324186f2cb (@127.0.0.1) 2.05ms
Apr 4 14:09:21 res-dev-jh-maths-vm bash[3774]: [I 2022-04-04 14:09:21.363 SingleUserLabApp log:189] 201 POST /user/jupyter-epo24/api/sessions?1649081359814 (jupyter-epo24@::ffff:138.38.220.241) 999.93ms
Apr 4 14:09:21 res-dev-jh-maths-vm bash[3432]: [I 2022-04-04 14:09:21.370 SingleUserLabApp log:189] 204 PUT /user/jupyter-eod34/lab/api/workspaces/default?1649081361881 (jupyter-eod34@::ffff:138.38.244.181) 3.54ms
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: 14:09:21.545 [ConfigProxy] #033[31merror#033[39m: 503 GET /user/jupyter-thh64/api/sessions?1649081357304 connect ECONNREFUSED 127.0.0.1:56899
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.547 JupyterHub pages:652] No template for 503
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.548 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fsessions%3F1649081357304 (@127.0.0.1) 2.02ms
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: 14:09:21.569 [ConfigProxy] #033[31merror#033[39m: 503 GET /user/jupyter-thh64/api/kernels?1649081357305 connect ECONNREFUSED 127.0.0.1:56899
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [D 2022-04-04 14:09:21.571 JupyterHub pages:652] No template for 503
Apr 4 14:09:21 res-dev-jh-maths-vm jupyterhub[1318]: [I 2022-04-04 14:09:21.572 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-thh64%2Fapi%2Fkernels%3F1649081357305 (@127.0.0.1) 1.74ms
Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! :hugs:
If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.
You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! :wave:
Welcome to the Jupyter community! :tada:
![welcome[bot] avatar](https://avatars.githubusercontent.com/in/4141?v=4 "Published by a pub.dev verified publisher"){kind=small}
I believe the odds are that the server in question is not running at the port in the configuration. This could be:
- it is running, but not at the expected port (database inconsistency)
- it is not running at all, but jupyterhub hasn't noticed (spawner poll bug)
Both should be fixed by stopping and starting the affected server(s).
If it persist across restart, then logs for the server itself and for jupyterhub surrounding the spawn event may help. It could be some configuration is causing it to listen on a port or ip other than the one that is expected.
Thanks @minrk - where can I find the jupyterhub/server logs? Can't see them.
Logs of the hub itself will depend on how you started it, but you posted them in the issue description already. If you are using systemdspawner, you can use journalctl to get logs of user servers:
journalctl -u jupyter-username
Thanks - I tried that but there is nothing in the logs for the user servers.
journalctl -u jupyter-*****
-- Logs begin at Thu 2021-04-22 15:17:25 UTC, end at Wed 2022-04-06 11:05:18 UTC. --
-- No entries --
I am still seeing this issue. Is there some cache somewhere for users that needs to be cleared?
The logs pulled out for an example user are:
[I 2022-10-19 10:55:57.719 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 1.55ms
[I 2022-10-19 10:55:58.388 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 1.75ms
[I 2022-10-19 10:56:25.572 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.98ms
[I 2022-10-19 10:56:25.984 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.14ms
[I 2022-10-19 10:56:26.151 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 1.66ms
[I 2022-10-19 10:56:26.310 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 1.92ms
[I 2022-10-19 10:56:26.427 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 1.86ms
[I 2022-10-19 10:56:26.559 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 1.68ms
[I 2022-10-19 10:57:28.456 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.28ms
[I 2022-10-19 10:57:28.848 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.07ms
[I 2022-10-19 10:57:29.009 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.34ms
[I 2022-10-19 10:57:29.826 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 1.54ms
[I 2022-10-19 10:57:29.984 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.32ms
[I 2022-10-19 10:57:30.153 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 1.69ms
[I 2022-10-19 10:57:30.315 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.57ms
[I 2022-10-19 10:57:30.801 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.73ms
[I 2022-10-19 10:57:35.271 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.83ms
[I 2022-10-19 10:57:36.919 JupyterHub login:44] User logged out: jupyter-sp2788
[I 2022-10-19 10:57:37.983 JupyterHub base:762] User logged in: jupyter-sp2788
[I 2022-10-19 10:57:37.984 JupyterHub log:189] 302 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] -> /user/jupyter-sp2788/ (@::ffff:138.38.235.163) 100.33ms
[I 2022-10-19 10:57:38.048 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 1.66ms
[I 2022-10-19 10:57:41.542 JupyterHub login:44] User logged out: jupyter-sp2788
[I 2022-10-19 10:57:44.868 JupyterHub base:762] User logged in: jupyter-sp2788
[I 2022-10-19 10:57:44.870 JupyterHub log:189] 302 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] -> /user/jupyter-sp2788/ (@::ffff:138.38.235.163) 98.70ms
[I 2022-10-19 10:57:44.935 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 2.14ms
[I 2022-10-19 10:59:11.456 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 1.74ms
[I 2022-10-19 10:59:11.835 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 1.56ms
[I 2022-10-19 10:59:11.995 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 2.44ms
[I 2022-10-19 10:59:16.993 JupyterHub login:44] User logged out: jupyter-sp2788
[I 2022-10-19 10:59:18.404 JupyterHub base:762] User logged in: jupyter-sp2788
[I 2022-10-19 10:59:18.405 JupyterHub log:189] 302 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] -> /user/jupyter-sp2788/ (@::ffff:138.38.235.163) 118.48ms
[I 2022-10-19 10:59:19.390 JupyterHub base:762] User logged in: jupyter-sp2788
[I 2022-10-19 10:59:19.391 JupyterHub log:189] 302 GET /hub/oauth_callback?code=[secret]&state=[secret]&session_state=[secret] -> /user/jupyter-sp2788/ (@::ffff:138.38.235.163) 151.75ms
[I 2022-10-19 10:59:19.455 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 2.76ms
[I 2022-10-19 11:03:33.392 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.40ms
[I 2022-10-19 11:03:36.102 JupyterHub log:189] 204 DELETE /hub/api/users/jupyter-sp2788/server (jupyter-sp2788@::ffff:138.38.235.163) 18.97ms
[I 2022-10-19 11:03:36.566 JupyterHub pages:148] User is running: jupyter-sp2788
[I 2022-10-19 11:03:36.567 JupyterHub log:189] 302 GET /hub/spawn/jupyter-sp2788 -> /user/jupyter-sp2788/ (jupyter-sp2788@::ffff:138.38.235.163) 3.94ms
[I 2022-10-19 11:03:36.593 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2F (@127.0.0.1) 2.13ms
[I 2022-10-19 11:03:38.958 JupyterHub log:189] 200 GET /hub/token (jupyter-sp2788@::ffff:138.38.235.163) 4.77ms
[I 2022-10-19 11:13:42.198 JupyterHub log:189] 200 GET /hub/error/503?url=%2Fuser%2Fjupyter-sp2788%2Ftree (@127.0.0.1) 2.06ms
Clicking the server restart has no effect