docker-stacks
docker-stacks copied to clipboard
jupyter
Using NB_UID produces, KeyError and error due to unwriteable home directory
Unwriteable home directory previously reported in PR #465. It was incorrectly thought that PR #555
addressed the issue. It did not, but it did introduce an additional error for this use case KeyError: 'runtime_dir'
[root@sandbox dropzone]# docker run --name sparkall --user root -e NB_USER=centos -e
NB_GROUP=centos -e NB_UID=$(id -u centos) -e NB_GID=$(id -g centos) -p 8888:8888 -v work:/home/centos/work jupyter/all-spark-notebook
Set username to: centos
Setting CWD to /home/centos/
Set centos GID to: 1000
Executing the command: jupyter notebook
Traceback (most recent call last):
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get
value = obj._trait_values[self.name]
KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-notebook", line 6, in <module>
sys.exit(notebook.notebookapp.main())
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 266, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 657, in launch_instance
app.initialize(argv)
File "<decorator-gen-7>", line 2, in initialize
File "/opt/conda/lib/python3.6/site-packages/traitlets/config/application.py", line 87, in catch_config_error
return method(app, *args, **kwargs)
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1505, in initialize
self.init_configurables()
File "/opt/conda/lib/python3.6/site-packages/notebook/notebookapp.py", line 1209, in init_configurables
connection_dir=self.runtime_dir,
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 556, in __get__
return self.get(obj, cls)
File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 535, in get
value = self._validate(obj, dynamic_default())
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/application.py", line 99, in _runtime_dir_default
ensure_dir_exists(rd, mode=0o700)
File "/opt/conda/lib/python3.6/site-packages/jupyter_core/utils/__init__.py", line 13, in ensure_dir_exists
os.makedirs(path, mode=mode)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 210, in makedirs
makedirs(head, mode, exist_ok)
File "/opt/conda/lib/python3.6/os.py", line 220, in makedirs
mkdir(name, mode)
PermissionError: [Errno 13] Permission denied: '/home/centos/.local'
The following shows permissions on the /home/centos directory (using ls -la instead of default CMD).
total 0
drwxr-xr-x 3 root root 18 Apr 2 20:58 .
drwxr-xr-x 4 root root 34 Apr 2 20:58 ..
drwxr-xr-x 2 root root 6 Mar 28 15:40 work
@harschware Can you try running the following and paste the output?
docker run --name sparkall --user root -e NB_USER=centos \
-e NB_GROUP=centos -e NB_UID=$(id -u centos) -e NB_GID=$(id -g centos) \
-p 8888:8888 -v work:/home/centos/work jupyter/all-spark-notebook start.sh ls -al /home/centos
The start.sh script is responsible for adjusting permissions, so I'd like to see what the permissions look like after the script performs its work.
[root@sandbox ~]# docker run --name sparkall --user root -e NB_USER=centos -e NB_GROUP=centos -e NB_UID=$(id -u centos) -e NB_GID=$(id -g centos) -p 8888:8888 -v work:/home/centos/work jupyter/all-spark-notebook start.sh ls -al /home/centos
Set username to: centos
Setting CWD to /home/centos/
Set centos GID to: 1000
Executing the command: ls -al /home/centos
total 0
drwxr-xr-x 3 root root 18 Apr 9 22:57 .
drwxr-xr-x 4 root root 34 Apr 9 22:57 ..
drwxr-xr-x 2 root root 6 Mar 28 15:40 work
On more question, can you paste the permissions of the work folder on the host that you are mounting into the container as /home/centos/work?
A couple of things I noticed:
-
if I don't specify a full path to -v as in "work:/home/centos/work" work will be created in /var/lib/docker/volumes by root.
[root@sandbox ~]# ll /var/lib/docker/volumes total 24 -rw------- 1 root root 32768 Apr 11 20:16 metadata.db drwxr-xr-x 3 root root 18 Apr 11 20:16 work
I mistakenly thought the volume "work" would be relative to the cwd of the host when "docker run" was executed.
2) if I pre-create a volume in cwd called work and change perms to 777 and use a -v of "$PWD/work:/home/centos/work" then the folder will show 777 when ls'd but still fails with PermissionError: [Errno 13] when the original docker run without start.sh is run. In other words permissions on the host don't seem to be the cause.
NOTE: you should be able to duplicate the issue in your environment if you choose a username on your host and use in place of 'centos' everywhere in the command. Be interesting to confirm it is an issue outside my env.
Have the same issue: docker pull jupyter/all-spark-notebook docker run -p 8888:8888 jupyter/all-spark-notebook
Error: Container must be run with group "root" to update passwd file Executing the command: jupyter notebook Traceback (most recent call last): File "/opt/conda/lib/python3.6/site-packages/traitlets/traitlets.py", line 528, in get value = obj._trait_values[self.name] KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/conda/bin/jupyter-notebook", line 11, in
Any advise?
@aie0 the no space left on device error is usually due to the Docker host hard drive being full. You can check by running df -h.
I am also still getting this same issue. Only getting it on a Docker host with Ubuntu 14.04 and Aufs storage driver... Is there a workaround to get it working there?
For a reproducible example (Dockerfile here):
docker run -d \
-e NB_UID=3006 \
-e NB_GID=3006 \
-e GRANT_SUDO=1 \
--user=root \
--name=jupyter-rad \
dnk8n/jupyter-rad:latest \
start-notebook.sh
However, running the image the above image is based off of instead:
docker run -d \
-e NB_UID=3006 \
-e NB_GID=3006 \
-e GRANT_SUDO=1 \
--user=root \
--name=jupyter-minimal \
jupyter/minimal-notebook:87210526f381 \
start-notebook.sh
does not produce the same error. It is potentially something to do with the device still being busy after chowning a whole lot more files and directories?
Spending the day reviewing and closing old issues, it's possible the original issue filed here could be related to https://github.com/jupyter/docker-stacks/issues/542#issuecomment-379515332.
I think the problem is caused by mapping external volume on first run.
If I run:
$ docker run -it --rm --user root -e NB_UID=1234 -e NB_GID=1234 -e NB_USER=username -e NB_GROUP=groupname jupyter/base-notebook start.sh
Set username to: username
Relocating home dir to /home/username
Setting CWD to /home/username/
Set username UID to: 1234
Add username to group: 1234
Executing the command: bash
username@9dbfc95b69ae:~$ pwd
/home/username
username@9dbfc95b69ae:~$ ls -la
total 48
drwsrwsr-x 8 username groupname 4096 Jul 4 09:05 .
drwxr-xr-x 1 root root 4096 Jul 8 13:03 ..
-rw-rw-r-- 1 username groupname 220 Apr 4 2018 .bash_logout
-rw-rw-r-- 1 username groupname 3770 Mar 14 02:54 .bashrc
drwsrwsr-x 2 username groupname 4096 Jul 4 09:05 .cache
drwsrwsr-x 2 username groupname 4096 Jul 4 08:55 .conda
drwsrws--- 3 username groupname 4096 Jul 4 09:02 .config
drwsrws--- 2 username groupname 4096 Jul 4 09:05 .jupyter
-rw-rw-r-- 1 username groupname 807 Apr 4 2018 .profile
drwsrwsr-x 2 username groupname 4096 Apr 8 23:37 work
drwsrwsr-x 3 username groupname 4096 Jul 4 09:04 .yarn
username@9dbfc95b69ae:~$
but if I run:
$ docker run -it --rm --user root -e NB_UID=1234 -e NB_GID=1234 -e NB_USER=username -e NB_GROUP=groupname -v `pwd`/test:/home/username/work jupyter/base-notebook start.sh
Set username to: username
Setting CWD to /home/username/
Set username UID to: 1234
Add username to group: 1234
Executing the command: bash
username@3bce498aebda:~$ pwd
/home/username
username@3bce498aebda:~$ ls -la
total 12
drwxr-xr-x 3 root root 4096 Jul 8 13:10 .
drwxr-xr-x 1 root root 4096 Jul 8 13:10 ..
drwxr-xr-x 2 1001 1001 4096 Jul 8 12:52 work
username@3bce498aebda:~$
So, looks like, when creating the new user's home directory if the home directory does not exists (as It was not externally mapped) the home directory is created with default files and permissions; but when the home directory exists (as It was created cause the external volume mapping -done by the root user-) then no default user files are created and directory permissions are not changed.
The only solution I found (without changing base images) is to create the container without the mapping (and without --rm), stop it, and re-run with the mapping (without --rm).
While upgrading from jupyterhub 1.0 to 1.2 we run into the same issue. The work-around for us was to change the docker entry-point and to manually change the permissions of some folders:
Our docker entry point now looks like this:
#!/bin/bash
if test -n "$NB_UID"; then
chown -R "$NB_UID":"$NB_GID" /home/jovyan/.local
chown -R "$NB_UID":"$NB_GID" /home/jovyan/.julia # locally installed julia packages
chown -R "$NB_UID":"$NB_GID" /home/jovyan/work # docker volume mount point
fi
exec /usr/local/bin/start-singleuser.sh --KernelSpecManager.ensure_native_kernel=False
It is my understand that in previous version, these permissions were changed automatically by the start-singleuser.sh script. In our case, the username is still jovyan but it should have a different user/group id (specified by $NB_UID and $NB_GID)
@Bidek56 I filed this issue 4 years ago and no longer work with jupyter code. I'm unsure the current status and lack the ability to test now. As far as I'm concerned the issue could be closed for being stale, so I'll close it.