help
help copied to clipboard
jupyter
permission denied when homes on NFS
Hi All,
I have successfully installed jupyterhub via conda after installing the conda-forge channel (thanks to everyone for that). I'm running on RHEL 6.7 with automounted NFS home directories for most users, these users exist in LDAP and are authenticated via pam.
When I try to log into jupyterhub using one of the LDAP authenticated, NFS Home directory users I get the error below. Please note, I have verified that logging in and all functions do work for local only users. Additionally, I created /nfs/home/userx/.local and gave it 777 permissions, still same error only the next time the error was for /nfs/home/userx/.local/shared.
Please let me know if there is any other information needed.
Thanks in advance.
Thomas
Command starting jupyterhub:
jupyterhub --no-ssl --log-level=DEBUG
Error: Spawning jupyterhub-singleuser --user=userx --port=37054 --cookie-name=jupyter-hub-token-userx --base-url=/user/userx --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1 [D 2016-06-12 15:47:32.676 JupyterHub spawner:316] Polling subprocess every 30s Traceback (most recent call last): File "/opt/anaconda3/lib/python3.5/site-packages/traitlets/traitlets.py", line 501, in get value = obj._trait_values[self.name] KeyError: 'runtime_dir'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 297, in
Hi Minrk,
I had already tried that, I stated it above but it was maybe mixed in. Additionally, I created ~/.local/shared "mkdir ~/.local/shared" and set permissions to 777, same permission denied errors as before.
Thanks
I think ~/.local/shared should be ~/.local/share (no d). But the underlying issue is that it's getting a permission error when trying to create directories. That sounds like something is wrong with your NFS setup, and creating each directory manually is not a great answer.
You are correct takluyver, it is share. I moved shared to share, same issue, now it can't create the jupyter folder. I turned off automount on this particular system and did a direct mount to home as such:
mount -o rw,nolock nfsserver:/home /nfs/home
Same issue.
Thanks
If you start python and manually run:
import os
os.mkdir('/nfs/home/userx/.local/share/jupyter')
Does that succeed, or fail with the same error?
And how about (after deleting the newly created directory):
import os
os.mkdir('/nfs/home/userx/.local/share/jupyter', mode=0o700)
That should be exactly the same call that's being made in Jupyter. Maybe it's not getting run correctly as the relevant user?
That was my thinking. I'm starting jupyterhub as root and as I understand it's supposed to spawn the notebooks as the requesting user and even in the output from the error above, it appears to be spawning as userx.
Is there something more that can be done to debug the issue? FYI, i'm not getting any input from the nfs logs indicating failed permissions or anything else for that matter.
Thanks
Back to you, @minrk - I'm not sure how to debug Jupyterhub spawner stuff.
Are you running these tests as the same user that you are trying to login
with JupyterHub? Can you start a regular notebook server as this user
(jupyter notebook)?
I wonder if there's something in the LDAP/PAM stuff that the Spawner's not switching to the right user. You could add some debug statements to check the user ID and name in the Spawner to make sure it's becoming the right user. I don't know how that could fail, but it would seem to produce what you are seeing.
-MinRK
Hi minrk,
I am able to run a jupyter notebook: $ jupyter notebook [I 09:12:21.900 NotebookApp] Writing notebook server cookie secret to /idn/home/tbisch/.local/share/jupyter/runtime/notebook_cookie_secret [I 09:12:22.024 NotebookApp] Serving notebooks from local directory: /idn/home/tbisch [I 09:12:22.024 NotebookApp] 0 active kernels [I 09:12:22.024 NotebookApp] The Jupyter Notebook is running at: http://localhost:8888/ [I 09:12:22.024 NotebookApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation). [W 09:12:22.030 NotebookApp] No web browser found: could not locate runnable browser.
When I authenticate and use the wrong password, it comes back and tells me as much so at least the LDAP authentication is working properly via jupyterhub and PAM. As for UID, it is only stored in LDAP so I would assume that it's handing over the proper ID. I did notice however, when logging in as a local user and looking at ps, the user was listed numerically instead of by username.
If you could tell me where and what to put in, I will add that for the additional debug to check spawner ID and name.
Thanks
Sorry, missed the first part, YES, running "jupyter notebook" as the same user as I'm trying to log into jupyterhub with.
Thanks
Hi All,
I've run into this situation once again and hoping someone may have an idea. Previously we were using LDAP to log into systems and uses that had their user id stored in LDAP were unable to log in with permission denied, never did figure out if it was due to nfs or whether it was caused by the LDAP login. We then switched the system to use Active Directory for authentication and magically it started working.
Unfortunately when I moved to the next machine I wanted to install jupyterhub to, I started getting the same/very similar error. The two systems should be identical, and definitely are identical as far as authenticating users with Active Directory and user permissions as the home directories are shared between the two machines.
Just to answer previously asked question:
- YES, as a user logged in authenticating against AD I can successfully do: "mkdir ~/.local" 2 YES, as a user logged in authenticating against AD I can successfully do: import os os.mkdir('/nfs/home/userx/.local/
- Yes, I am running all tests as the same user that is getting the error from jupyterhub
- YES, as a user logged in authenticating against AD I can successfully run: /opt/anaconda3/bin/jupyter notebook
Two additional piece of info is that I created a local user, that had a local home directory on this new system and everything works fine. I then changed the home directory to the nfs mount, sent ownership accordingly and that also worked.
Our users in Active directory have unusually high UIDs, for instance 73458987 is one user id, not sure if that could be causing an issue.
Once again, here is the error I see when running the following:
# jupyterhub --no-ssl --log-level=DEBUG
[D 2016-08-03 14:50:42.850 JupyterHub application:529] Looking for jupyterhub_config in None
[I 2016-08-03 14:50:42.851 JupyterHub app:622] Loading cookie_secret from /root/jupyterhub_cookie_secret
[D 2016-08-03 14:50:42.852 JupyterHub app:694] Connecting to db: sqlite:///jupyterhub.sqlite
[W 2016-08-03 14:50:42.889 JupyterHub app:304]
Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.
[W 2016-08-03 14:50:42.894 JupyterHub app:757] No admin users, admin interface will be unavailable.
[W 2016-08-03 14:50:42.894 JupyterHub app:758] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2016-08-03 14:50:42.894 JupyterHub app:785] Not using whitelist. Any authenticated user will be allowed.
[D 2016-08-03 14:50:42.910 JupyterHub app:888] Loaded users:
aduser1
[I 2016-08-03 14:50:42.920 JupyterHub app:1231] Hub API listening on http://127.0.0.1:8081/hub/
[W 2016-08-03 14:50:42.924 JupyterHub app:959] Running JupyterHub without SSL. There better be SSL termination happening somewhere else...
[I 2016-08-03 14:50:42.925 JupyterHub app:968] Starting proxy @ http://*:8000/
[D 2016-08-03 14:50:42.925 JupyterHub app:969] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--default-target', 'http://127.0.0.1:8081', '--error-target', 'http://127.0.0.1:8081/hub/error']
14:50:43.140 - info: [ConfigProxy] Proxying http://*:8000 to http://127.0.0.1:8081
14:50:43.145 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[D 2016-08-03 14:50:43.233 JupyterHub app:997] Proxy started and appears to be up
[I 2016-08-03 14:50:43.233 JupyterHub app:1254] JupyterHub is now running at http://127.0.0.1:8000/
[I 2016-08-03 14:50:48.602 JupyterHub log:100] 302 GET / (@10.1.23.02) 1.75ms
[I 2016-08-03 14:50:48.648 JupyterHub log:100] 302 GET /hub (@10.1.23.02) 0.46ms
[I 2016-08-03 14:50:48.705 JupyterHub log:100] 302 GET /hub/ (@10.1.23.02) 0.78ms
[I 2016-08-03 14:50:48.793 JupyterHub log:100] 200 GET /hub/login (@10.1.23.02) 38.55ms
[D 2016-08-03 14:50:49.133 JupyterHub log:100] 304 GET /favicon.ico (@10.1.23.02) 8.41ms
[I 2016-08-03 14:51:02.277 JupyterHub spawner:467] Spawning jupyterhub-singleuser --user=aduser1 --port=44561 --cookie-name=jupyter-hub-token-aduser1 --base-url=/user/aduser1 --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1
[D 2016-08-03 14:51:02.395 JupyterHub spawner:316] Polling subprocess every 30s
[W 2016-08-03 14:51:02.755 aduser1 loader:419] Unrecognized JSON config file version, assuming version 1
Using Anaconda Cloud api site https://api.anaconda.org
[I 2016-08-03 14:51:03.650 aduser1 manager:21] [nb_conda_kernels] enabled, 1 kernels found
[I 2016-08-03 14:51:04.104 aduser1 handlers:250] [nb_conda] enabled
[I 2016-08-03 14:51:04.164 aduser1 handlers:73] [nb_anacondacloud] enabled
[I 2016-08-03 14:51:04.238 aduser1 __init__:35] \u2713 nbpresent HTML export ENABLED
[W 2016-08-03 14:51:04.238 aduser1 __init__:43] \u2717 nbpresent PDF export DISABLED: No module named 'nbbrowserpdf'
[I 2016-08-03 14:51:04.246 aduser1 notebookapp:1128] Serving notebooks from local directory: /mnt/home/aduser1
[I 2016-08-03 14:51:04.246 aduser1 notebookapp:1128] 0 active kernels
[I 2016-08-03 14:51:04.246 aduser1 notebookapp:1128] The Jupyter Notebook is running at: http://127.0.0.1:44561/user/aduser1/
[I 2016-08-03 14:51:04.247 aduser1 notebookapp:1129] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Traceback (most recent call last):
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 297, in <module>
main()
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 293, in main
return SingleUserNotebookApp.launch_instance()
File "/opt/anaconda3/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/traitlets/config/application.py", line 596, in launch_instance
app.start()
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 253, in start
super(SingleUserNotebookApp, self).start()
File "/opt/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 1131, in start
self.write_server_info_file()
File "/opt/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 1105, in write_server_info_file
with open(self.info_file, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/mnt/home/aduser1/.local/share/jupyter/runtime/nbserver-17191.json'
[W 2016-08-03 14:51:12.496 JupyterHub web:1521] 500 POST /hub/login?next= (10.1.23.02): Spawner failed to start [status=1]
[D 2016-08-03 14:51:12.499 JupyterHub base:441] No template for 500
[E 2016-08-03 14:51:12.511 JupyterHub log:99] {
"Connection": "close",
"Content-Type": "application/x-www-form-urlencoded",
"X-Forwarded-Port": "8000",
"Origin": "http://192.16.151.132:8000",
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8",
"Accept-Encoding": "gzip, deflate",
"Accept-Language": "en-US,en;q=0.8",
"Upgrade-Insecure-Requests": "1",
"X-Forwarded-Proto": "http",
"Content-Length": "38",
"X-Forwarded-For": "10.1.23.02",
"Referer": "http://192.16.151.132:8000/hub/login",
"Cache-Control": "max-age=0",
"Host": "192.16.151.132:8000"
}
[E 2016-08-03 14:51:12.511 JupyterHub log:100] 500 POST /hub/login?next= (@10.1.23.02) 13781.08ms
^C
Interrupted
[I 2016-08-03 14:51:17.798 JupyterHub app:1115] Cleaning up single-user servers...
[I 2016-08-03 14:51:18.041 JupyterHub app:1126] Cleaning up proxy[16779]...
[I 2016-08-03 14:51:18.041 JupyterHub app:1152] ...done
I ran strace on the process and have attached it, hopefully someone can shed some insight.
Thanks in advance.
That exception looks like the user doesn't own their own home directory. This could be because the user has not received the correct UID, or it could be that the something has gone awry with the permissions on that directory from a previous run.
If the user's jupyter_notebook_config.py, you can add some debugging information, such as:
import os, getpass
print('uid=%i' % os.getuid())
print('user=%s' % getpass.getuser())
to see if the username and uid are what you expect them to be.
You can also check the permissions on /mnt/home/aduser1/.local/share/jupyter/runtime and its parents.
Hi Minrk,
I am unable to find any file jupyter_notebook_config.py anywhere on the system, including under .local in the user's home directory.
/mnt/home/aduser1 is owned by aduser1 and perms are 750. That being said, I should be able to delete .local, and when I log into jupyterhub, it should all be recreated.
Note, I have one system, using shared NFS directories and AD auth, where jupyterhub works just fine, /mnt/home/some_user/.local is created as expected upon logging in and notebooks are spawned fine. Two other systems, with the exact same config, using the same AD authentication server and the same NFS mounted home directories, fails with the permission denied.
Thanks
@mymlact sorry. You can create an empty notebook config file with jupyter notebook --generate-config, and edit ~/.jupyter/jupyter_notebook_config.py. Or you can create that file by hand with the above contents.
I created the config file and it returned the correct user ID and UID.
# jupyterhub --no-ssl --log-level=DEBUG
[D 2016-08-05 12:32:40.042 JupyterHub application:529] Looking for jupyterhub_config in None
[I 2016-08-05 12:32:40.048 JupyterHub app:622] Loading cookie_secret from /mnt/home/aduser1/.jupyter/jupyterhub_cookie_secret
[D 2016-08-05 12:32:40.050 JupyterHub app:694] Connecting to db: sqlite:///jupyterhub.sqlite
[W 2016-08-05 12:32:40.115 JupyterHub app:304]
Generating CONFIGPROXY_AUTH_TOKEN. Restarting the Hub will require restarting the proxy.
Set CONFIGPROXY_AUTH_TOKEN env or JupyterHub.proxy_auth_token config to avoid this message.
[W 2016-08-05 12:32:40.125 JupyterHub app:757] No admin users, admin interface will be unavailable.
[W 2016-08-05 12:32:40.125 JupyterHub app:758] Add any administrative users to `c.Authenticator.admin_users` in config.
[I 2016-08-05 12:32:40.125 JupyterHub app:785] Not using whitelist. Any authenticated user will be allowed.
[D 2016-08-05 12:32:40.152 JupyterHub app:888] Loaded users:
aduser1
[I 2016-08-05 12:32:40.168 JupyterHub app:1231] Hub API listening on http://127.0.0.1:8081/hub/
[W 2016-08-05 12:32:40.180 JupyterHub app:959] Running JupyterHub without SSL. There better be SSL termination happening somewhere else...
[I 2016-08-05 12:32:40.180 JupyterHub app:968] Starting proxy @ http://*:8000/
[D 2016-08-05 12:32:40.181 JupyterHub app:969] Proxy cmd: ['configurable-http-proxy', '--ip', '', '--port', '8000', '--api-ip', '127.0.0.1', '--api-port', '8001', '--default-target', 'http://127.0.0.1:8081', '--error-target', 'http://127.0.0.1:8081/hub/error']
12:32:40.394 - info: [ConfigProxy] Proxying http://*:8000 to http://127.0.0.1:8081
12:32:40.399 - info: [ConfigProxy] Proxy API at http://127.0.0.1:8001/api/routes
[D 2016-08-05 12:32:40.488 JupyterHub app:997] Proxy started and appears to be up
[I 2016-08-05 12:32:40.488 JupyterHub app:1254] JupyterHub is now running at http://127.0.0.1:8000/
[I 2016-08-05 12:32:44.709 JupyterHub log:100] 302 GET / (@192.16.151.132) 2.90ms
[I 2016-08-05 12:32:44.714 JupyterHub log:100] 302 GET /hub (@192.16.151.132) 0.46ms
[I 2016-08-05 12:32:44.734 JupyterHub log:100] 302 GET /hub/ (@192.16.151.132) 1.55ms
[I 2016-08-05 12:32:44.788 JupyterHub log:100] 200 GET /hub/login (@192.16.151.132) 39.64ms
[D 2016-08-05 12:32:44.922 JupyterHub log:100] 304 GET /favicon.ico (@192.16.151.132) 8.30ms
[I 2016-08-05 12:32:53.027 JupyterHub spawner:467] Spawning jupyterhub-singleuser --user=aduser1 --port=40382 --cookie-name=jupyter-hub-token-aduser1 --base-url=/user/aduser1 --hub-host= --hub-prefix=/hub/ --hub-api-url=http://127.0.0.1:8081/hub/api --ip=127.0.0.1
[D 2016-08-05 12:32:53.048 JupyterHub spawner:316] Polling subprocess every 30s
[W 2016-08-05 12:32:53.513 aduser1 loader:419] Unrecognized JSON config file version, assuming version 1
uid=73093813
user=aduser1
Using Anaconda Cloud api site https://api.anaconda.org
[I 2016-08-05 12:32:54.393 aduser1 manager:21] [nb_conda_kernels] enabled, 1 kernels found
[I 2016-08-05 12:32:54.910 aduser1 handlers:73] [nb_anacondacloud] enabled
[I 2016-08-05 12:32:54.988 aduser1 __init__:35] \u2713 nbpresent HTML export ENABLED
[W 2016-08-05 12:32:54.989 aduser1 __init__:43] \u2717 nbpresent PDF export DISABLED: No module named 'nbbrowserpdf'
[I 2016-08-05 12:32:54.994 aduser1 handlers:250] [nb_conda] enabled
[I 2016-08-05 12:32:55.002 aduser1 notebookapp:1128] Serving notebooks from local directory: /mnt/home/aduser1
[I 2016-08-05 12:32:55.002 aduser1 notebookapp:1128] 0 active kernels
[I 2016-08-05 12:32:55.002 aduser1 notebookapp:1128] The Jupyter Notebook is running at: http://127.0.0.1:40382/user/aduser1/
[I 2016-08-05 12:32:55.003 aduser1 notebookapp:1129] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
Traceback (most recent call last):
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 297, in <module>
main()
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 293, in main
return SingleUserNotebookApp.launch_instance()
File "/opt/anaconda3/lib/python3.5/site-packages/jupyter_core/application.py", line 267, in launch_instance
return super(JupyterApp, cls).launch_instance(argv=argv, **kwargs)
File "/opt/anaconda3/lib/python3.5/site-packages/traitlets/config/application.py", line 596, in launch_instance
app.start()
File "/opt/anaconda3/bin/jupyterhub-singleuser", line 253, in start
super(SingleUserNotebookApp, self).start()
File "/opt/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 1131, in start
self.write_server_info_file()
File "/opt/anaconda3/lib/python3.5/site-packages/notebook/notebookapp.py", line 1105, in write_server_info_file
with open(self.info_file, 'w') as f:
PermissionError: [Errno 13] Permission denied: '/mnt/home/aduser1/.local/share/jupyter/runtime/nbserver-22461.json'
[W 2016-08-05 12:33:03.092 JupyterHub web:1521] 500 POST /hub/login?next= (192.16.151.132): Spawner failed to start [status=1]
[D 2016-08-05 12:33:03.101 JupyterHub base:441] No template for 500
[E 2016-08-05 12:33:03.122 JupyterHub log:99] {
"Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
"Content-Type": "application/x-www-form-urlencoded",
"User-Agent": "Mozilla/5.0 (X11; OpenBSD amd64; rv:47.0) Gecko/20100101 Firefox/47.0",
"Accept-Language": "en-US,en;q=0.5",
"X-Forwarded-For": "192.16.151.132",
"Connection": "close",
"X-Forwarded-Proto": "http",
"Accept-Encoding": "gzip, deflate",
"Host": "system56.mydomain.com:8000",
"Content-Length": "38",
"X-Forwarded-Port": "8000",
"Referer": "http://system56.mydomain.com:8000/hub/login"
}
[E 2016-08-05 12:33:03.122 JupyterHub log:100] 500 POST /hub/login?next= (@192.16.151.132) 10314.18ms
[D 2016-08-05 12:33:03.235 JupyterHub log:100] 304 GET /favicon.ico (@192.16.151.132) 0.85ms
[W 2016-08-05 12:33:23.128 JupyterHub user:264] aduser1's server never showed up at http://127.0.0.1:40382/user/aduser1 after 30 seconds. Giving up
[E 2016-08-05 12:33:23.173 JupyterHub gen:871] Exception in Future <tornado.concurrent.Future object at 0x7f357a1ca128> after timeout
Traceback (most recent call last):
File "/opt/anaconda3/lib/python3.5/site-packages/tornado/gen.py", line 867, in error_callback
future.result()
File "/opt/anaconda3/lib/python3.5/site-packages/jupyterhub/user.py", line 280, in spawn
raise e
File "/opt/anaconda3/lib/python3.5/site-packages/jupyterhub/user.py", line 256, in spawn
yield self.server.wait_up(http=True, timeout=spawner.http_timeout)
File "/opt/anaconda3/lib/python3.5/site-packages/jupyterhub/orm.py", line 108, in wait_up
yield wait_for_http_server(self.url, timeout=timeout)
File "/opt/anaconda3/lib/python3.5/site-packages/jupyterhub/utils.py", line 94, in wait_for_http_server
**locals()
TimeoutError: Server at http://127.0.0.1:40382/user/aduser1 didn't respond in 30 seconds
[D 2016-08-05 12:37:40.514 JupyterHub orm:146] Fetching GET http://127.0.0.1:8001/api/routes
[D 2016-08-05 12:42:40.531 JupyterHub orm:146] Fetching GET http://127.0.0.1:8001/api/routes
^C
Interrupted
[I 2016-08-05 12:44:31.684 JupyterHub app:1115] Cleaning up single-user servers...
[I 2016-08-05 12:44:31.737 JupyterHub app:1126] Cleaning up proxy[22204]...
[I 2016-08-05 12:44:31.738 JupyterHub app:1152] ...done
This really makes no sense, especially since it works perfectly from a different jupyterhub host mounting the same NFS home directory using the same pam configuration and same AD server for authentication. I can even delete the .local directory on one system, fire up jupyterhub and log right in, creates the directories as expected.
Any other ideas?
Thanks.
Can you make the directory by hand (as aduser1) on the system where it doesn't work?
mkdir -p /mnt/home/aduser1/.local/share/jupyter/runtime
echo 'test' > /mnt/home/aduser1/.local/share/jupyter/runtime/test
Hi Minrk,
Yes, I am able to create files and directories in the NFS shared home , as the aduser1 user, on both the system where jupyterhub works as well as the one where jupyterhub does not work.
Specifically, both of the above commands work.
Thanks
I have the same issue , did someone used nfs shares from jupyterhub with ad credentials and fixed the access issues?
