Matt Riedemann
These are the packages we have installed FWIW: ``` attrs==21.4.0,awesome-progress-bar==1.7.2,certifi==2021.10.8,cffi==1.15.0,charset-normalizer==2.0.12,Deprecated==1.2.13,ibmq-deploy==1.14.2,idna==3.3,importlib-resources==5.7.1,jsonschema==4.5.0,pycparser==2.21,PyGithub==1.55,PyJWT==2.3.0,PyNaCl==1.5.0,pyrsistent==0.18.1,PyYAML==6.0,requests==2.27.1,ruamel.yaml==0.17.21,ruamel.yaml.clib==0.2.6,urllib3==1.26.9,wrapt==1.14.1,zipp==3.8.0 ```
Running our unit tests also hangs. I killed the test runner and got this output, it looks like there is maybe a cycle in here? ``` ========================================================================================== ERRORS =========================================================================================== ______________________________________________________________________...
This is the file that defines our schema: ```python """The json schema for the config file.""" STRING = {"type": "string"} BOOL = {"type": "boolean"} DEPLOY_SCHEMA = { "type": "object", "properties":...
> it'd also be helpful if you minimized that to the smallest hanging example I'm not sure what you mean. We had some code that hung validating that exact yaml...
> Moving to 4.5.1 fixes it. Same here, thanks @Julian.
I realize that latest z2jh is using kube-scheduler 1.19.7 so that would fix the issue for us if we upgraded to the latest z2jh, but if possible it'd be good...
If a 0.10.7 release isn't an option then maybe at least calling out the known issue in the docs is an alternative.
> Is the only required change for 0.10.6 `scheduling.userScheduler.image.tag: v1.19.5` Yes, in this case. I also ran into https://github.com/jupyterhub/kubespawner/issues/466 but that's a different issue which is already fixed in kubespawner...
FWIW, my second run of the scale test, to go from 1K to 2K active user notebook pods, started hitting failures around 1830 total active user pods. The hub restarted...
> @mriedem does `kubectl logs deploy/hub --previous` work or not? Got the logs back, it was what I suspected, consecutive spawn failure limit being reached: >Feb 5 11:02:21 hub-85499595cb-46b7s hub...