zero-to-jupyterhub-k8s
zero-to-jupyterhub-k8s copied to clipboard
jupyterhub
Add information about using NFS with z2jh
NFS is still a very popular storage setup, and is a good fit for use with z2jh in several cases:
- When you are supporting a large number of users
- When you are running on baremetal and NFS is your only option
- When your utilization % (% of total users active at any time) is very low, causing you to spend more on storage than compute.
While we don't want to be on the hook for teaching users to setup and maintain NFS servers, we should document how to use an NFS serer that already exists.
@yuvipanda I'd like to be a guinea pig on this. I am trying to setup a persistent EFS volume and use that as the user storage.
So far I've created and applied:
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-persist
spec:
capacity:
storage: 123Gi
accessModes:
- ReadWriteMany
nfs:
server: <fs-id>.efs.us-east-1.amazonaws.com
path: "/"
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: efs-persist
spec:
accessModes:
- ReadWriteMany
resources:
requests:
storage: 11Gi
After that I added the following to my config.yaml:
singleuser:
storage:
static:
pvc-name: efs-persist
I am pretty sure I am missing a few key ideas here
Edit: first change was to add "type: static" to the storage section in config changed pvc-name to pvcName
w00t, thanks for volunteering :)
The other two things to keep in mind are:
-
subPath(https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/master/jupyterhub/values.yaml#L138). This specifies where inside the share the user's homedirectory should go. Although it defaults to just{username}, I would recommend something likehome/{username} - User permissions. This can be a little tricky, since IIRC when kubelet creates a directory for subPath mounting it makes it uid 0 / gid 0, which is problematic for our users (with uid 1000 by default). The way we've worked around it right now is by using anongid / anonuid properties in our NFS share, but that's not a good long term solution. I've been working on http://github.com/yuvipanda/nfs-flex-volume as another option here. Is anongid / anonuid an option with EFS?
- I saw that, thanks for the clearer explanation
- I don't think that anonuid/gid is an option on EFS
I'll take a look through the nfs-flex-volume repo
@cam72cam another way to check that everything works right now except for permissions is to set:
singleuser:
uid: 0
fsGid: 0
cmd:
- jupyterhub-singleuser
- --allow-root
If you can launch servers with that, then we can confirm that the uid situation is the only problem.
I am currently getting the following response:
HTTP response body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Pod in version \"v1\" cannot be handled as a Pod: v1.Pod: Spec: v1.PodSpec: Containers: []v1.Container: v1.Container: VolumeMounts: []v1.VolumeMount: v1.VolumeMount: SubPath: ReadString: expects \" or n, parsing 184 ...ubPath\": {... at {\"kind\": \"Pod\", \"spec\": {\"containers\": [{\"imagePullPolicy\": \"IfNotPresent\", \"lifecycle\": {}, \"ports\": [{\"containerPort\": 8888, \"name\": \"notebook-port\"}], \"volumeMounts\": [{\"subPath\": {\"username\": null}, \"name\": \"home\", \"mountPath\": \"/home/jovyan\"}, {\"readOnly\": true, \"name\": \"no-api-access-please\", \"mountPath\": \"/var/run/secrets/kubernetes.io/serviceaccount\"}], \"env\": [{\"name\": \"JUPYTERHUB_HOST\", \"value\": \"\"}, {\"name\": \"JUPYTERHUB_CLIENT_ID\", \"value\": \"user-efs4\"}, {\"name\": \"JUPYTERHUB_API_TOKEN\", \"value\": \"355dc09aca1143f580ee0435339cc18d\"}, {\"name\": \"JUPYTERHUB_USER\", \"value\": \"efs4\"}, {\"name\": \"EMAIL\", \"value\": \"efs4@local\"}, {\"name\": \"GIT_AUTHOR_NAME\", \"value\": \"efs4\"}, {\"name\": \"JUPYTERHUB_ADMIN_ACCESS\", \"value\": \"1\"}, {\"name\": \"JUPYTERHUB_SERVICE_PREFIX\", \"value\": \"/user/efs4/\"}, {\"name\": \"JPY_API_TOKEN\", \"value\": \"355dc09aca1143f580ee0435339cc18d\"}, {\"name\": \"JUPYTERHUB_API_URL\", \"value\": \"http://100.65.96.26:8081/hub/api\"}, {\"name\": \"JUPYTERHUB_BASE_URL\", \"value\": \"/\"}, {\"name\": \"JUPYTERHUB_OAUTH_CALLBACK_URL\", \"value\": \"/user/efs4/oauth_callback\"}, {\"name\": \"GIT_COMMITTER_NAME\", \"value\": \"efs4\"}, {\"name\": \"MEM_GUARANTEE\", \"value\": \"1073741824\"}], \"image\": \"jupyterhub/k8s-singleuser-sample:v0.5.0\", \"resources\": {\"limits\": {}, \"requests\": {\"memory\": 1073741824}}, \"args\": [\"jupyterhub-singleuser\", \"--ip=\\\"0.0.0.0\\\"\", \"--port=8888\"], \"name\": \"notebook\"}], \"securityContext\": {\"runAsUser\": 1000, \"fsGroup\": 1000}, \"volumes\": [{\"persistentVolumeClaim\": {\"claimName\": \"efs-persist\"}, \"name\": \"home\"}, {\"emptyDir\": {}, \"name\": \"no-api-access-please\"}], \"initContainers\": []}, \"metadata\": {\"labels\": {\"hub.jupyter.org/username\": \"efs4\", \"heritage\": \"jupyterhub\", \"component\": \"singleuser-server\", \"app\": \"jupyterhub\"}, \"name\": \"jupyter-efs4\"}, \"apiVersion\": \"v1\"}","reason":"BadRequest","code":400}
I suspect it has to do with:
'volumes': [{'name': 'home',
'persistentVolumeClaim': {'claimName': 'efs-persist'}},
{'aws_elastic_block_store': None,
'azure_disk': None,
'azure_file': None,
'cephfs': None,
'cinder': None,
'config_map': None,
'downward_api': None,
'empty_dir': {},
'fc': None,
'flex_volume': None,
'flocker': None,
'gce_persistent_disk': None,
'git_repo': None,
'glusterfs': None,
'host_path': None,
'iscsi': None,
'name': 'no-api-access-please',
'nfs': None,
'persistent_volume_claim': None,
'photon_persistent_disk': None,
'portworx_volume': None,
'projected': None,
'quobyte': None,
'rbd': None,
'scale_io': None,
'secret': None,
'storageos': None,
'vsphere_volume': None}]},
~~It should have the nfs option set there if I understand correctly~~
Actually:
'volume_mounts': [{'mountPath': '/home/jovyan',
'name': 'home',
'subPath': {'username': None}},
{'mount_path': '/var/run/secrets/kubernetes.io/serviceaccount',
'name': 'no-api-access-please',
'read_only': True,
'sub_path': None}],
it appears that subpath is not being set correctly
So it turns out specifying subPath manually of home/{username} was required, so we should investigate why.
The PVC needs to be in the same namespace as JupyterHub, so the pods can find it.
The PVC needs to be told how to find the PV to match, and this is done by using:
- Labels to match PVC and PV
- Setting storageclass of PVC to '' (so kubernetes does not try to create a PV for it)
So
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-persist
labels:
<some-label1-key>: <some-label1-value>
<some-label2-key>: <some-label2-value>
spec:
capacity:
storage: 123Gi
accessModes:
- ReadWriteMany
nfs:
server: <fs-id>.efs.us-east-1.amazonaws.com
path: "/"
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: efs-persist
spec:
accessModes:
- ReadWriteMany
storageClassName: ""
selector:
matchLabels:
<some-label1-key>: <some-label1-value>
<some-label2-key>: <some-label2-value>
resources:
requests:
storage: 11Gi
Things to note:
- Specify labels that uniquely identify this PV in your entire k8s cluster.
- The size requests bits are ignored both in the PV and PVC for EFS specifically, since it grows as you use it.
Ok, I've got the mount working. I did not do the label stuff yet, simply set 'storageClassName: ""' in the claim. That seemed to work just fine.
I ran into a speed bump where I had to change the security groups to allow access from EC2 to EFS. As a temporary measure I added both the EFS volume and the EC2 instances to the "default" security group. Eventually part of the initial kops config should add the correct security groups.
I am now getting a permission error: PermissionError: [Errno 13] Permission denied: '/home/jovyan/.jupyter'
I am going to try to change the permissions on the EFS drive first, and if that does not work try the root hack that @yuvipanda mentioned
EDIT: A manual chown on the EFS volume to 1000:1000 seems to have worked!
EFS Success!
Process:
Setup an EFS volume. It must be in the same VPC as your cluster. This can be changed in the AWS settings after it has been created. The EFS volume will be created in the default security group in the VPC. As a temporary hack around, add your cluster master and nodes to the default VPC group so they can access the EFS volume. Eventually we will setup proper security groups as part of this process.
Created test_efs.yaml
apiVersion: v1
kind: PersistentVolume
metadata:
name: efs-persist
spec:
capacity:
storage: 123Gi
accessModes:
- ReadWriteMany
nfs:
server: fs-$EFS_ID.efs.us-east-1.amazonaws.com
path: "/"
Created test_efs_claim.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
name: efs-persist
spec:
storageClassName: ""
accessModes:
- ReadWriteMany
resources:
requests:
storage: 11Gi
kubectl --namespace=cmesh-test apply -f test_efs.yaml kubectl --namespace=cmesh-test apply -f test_efs_claim.yaml
The sizes in these files don't mean what you think. There is no quota enforced with EFS **. In the future we want to set the efs PersistentVolume size to something ridiculously large like 8Ei and the PersistentVolumeClaim to 10GB (neither matters AFAICT). This is my rough understanding and could be incorrect.
A PersistentVolume defines a service which can perform a mount inside of a container. The PersistentVolumeClaim is a way of reserving a portion of the PersistentVolume and potentially locking access to it.
The storageClassName setting looks innocuous, but it is incredibly critical. The only non storage class PV in the cluster is the one we defined above. In the future we should tag different PV's and use tag filters in the PVC instead of relying on a default of "".
We are going to configure jupyterhub to use the same "static" claim among all of the containers*** . This means that all of our users will be using the same EFS share which should be able to scale as high as we need.
We now add the following to config.yaml
singleuser:
storage:
type: "static"
static:
pvcName: "efs-persist"
subPath: 'home/{username}'
type static tells jh not to use a storage class and instead use a PVC defined below. pvcName matches the claim name we specified before subPath tells where on the supplied storage the mount point should be. In this case it will be "$EFS_ROOT/home/{username}"
It turns out there is a bug in jupyterhub where the default subPath does not work, and setting the subPath to "{username}" breaks in the same way.
At this point if we tried to start our cluster, it would fail. The directory created on the mount at subPath will be created with uid:0 and gid:0. This means that when jupyter hub is launched it won't be able to create any files, will complain, then self destruct.
What we need to do is tell the container to run our jupyterhub setup as root, then switch to the Jovian user before starting the jupyterhub process. When we are running as root we can do our own chown to adjust the created directory permissions.
First we merge the following to our config.yaml
singleuser:
uid: 0
fsGid: 0
cmd: "start-singleuser.sh"
This tells jupyterhub to enter the container as root and run the start-singleuser script. Start-singleuser calls a helper start.sh script which we will use later on.
This will get jupyter hub to provision the container and attempt to start, but the process will still fail as the chmod has not taken place.
In order for us to have a properly chowned directory in /home/Jovian mounted from $EFS_ROOT/home/{username}, we need to create our own docker container***.
Here are some terse steps: Create a docker account Create a docker repo Create a directory to store the build file Create Dockerfile inside that directory
FROM jupyter/base-notebook:281505737f8a
# pin jupyterhub to match the Hub version
# set via --build-arg in Makefile
ARG JUPYTERHUB_VERSION=0.8
RUN pip install --no-cache jupyterhub==$JUPYTERHUB_VERSION
USER root
RUN sed -i /usr/local/bin/start.sh -e 's,# Handle username change,chown 1000:1000 /home/$NB_USER \n # Handle username change,'
RUN cat /usr/local/bin/start.sh
USER $NB_USER
The base dockerfile came from https://github.com/jupyterhub/zero-to-jupyterhub-k8s/commit/967b2d2a2c6293ba686c8e57a9f9473575c1494e#diff-aed8b29ee8beb1247469956c481040c2 Notice that we are using the older revision. The newer revision is broken in some awesome way that Yuvi needs to fix. This script is fragile and should be done better in the future... The first parts of the docker setup are done as $NB_USER The rest of the file is done as ROOT since we need to modify the start.sh script which will be run as root when the container is started. Many of the files referenced can be found in https://github.com/jupyter/docker-stacks/tree/master/base-notebook sudo yum install docker sudo docker login sudo docker build ${directory_containing_dockerfile} sudo docker tag ${image_id_in_cmd_output} $docker_username/$docker_repo # can also be found by sudo docker images sudo docker push $docker_username/$docker_repo
Merge the following into config.yaml
singleuser:
image:
name: $docker_username/$docker_repo
tag: latest
You may be able to do a helm upgrade, but I ended up purging and reinstalling via helm just to be safe.
At this point you should be all set with a semi-fragile (but functional) EFS backed jupyterhub setup
Debugging tools: (all with --namespace=
- kubectl get pods # list pods
- kubectl logs $podname # get a pod log, may not be anything if the crash happens soon enough
- kubectl describe pod $podname # dumps a bunch of useful info about the pod
- kubectl get pod $podname -o yaml # dumps the args used to create the pod. This stuff is the container creation stuff
** fuse layer for fs quota
*** We may run into issues with a hundred containers all hitting the same EFS volume. I suspect that AWS can more than handle that, but I have been wrong before. If it can't handle that Yuvi has a WIP nfs server sharding system partially built that we could use.
**** I hope that the changes I made to the base container will be adopted by the project as it seems relatively harmless to have in the start script. Even if it is harmful to others, I would still like it in there as a config option (if possible).
Thank you for getting this working, @cam72cam!
To summarize, this mostly works, except for the issue of permissions:
- When using
subPath, Kubernetes creates this directory when it doesn't exist if it needs to - However, this will always be created as root:root
- Since we want our users to run as non-root, this won't work for us and we have to use hacks to do chowns.
It'll be great if we can fix EFS or Kubernetes to have options around 'what user / group / mode should this directory be created as?'
Could we add the chown hack I put in my image to the start.sh script in the stacks repo? https://github.com/jupyter/docker-stacks/blob/master/base-notebook/start.sh#L19
Would that break any existing users setups?
There are a few issues in Kubernetes which seem to disagree on whether Kubernetes should set the permissions on subpaths or not:
- https://github.com/kubernetes/kubernetes/issues/39474
- https://github.com/kubernetes/kubernetes/issues/41638
- https://github.com/kubernetes/kubernetes/pull/43775
I figure doing the chown ourselves resolves it for now (behind a config setting) and can be removed once K8S finalizes if/how/when the subpath permissions should be set
singleuser:
image:
name: jupyter/base-notebook
tag: 29b68cd9e187
extraEnv:
CHOWN_HOME: 'yes'
Just confirmed the fix in my test env
I was able to use my NFS-backed persistent claim on Google Cloud as the user storage by following the steps @cam72cam outlined, so I can attest his solution. Thanks for paving the way guys!
Just to clarify, do we do a helm installation first using a config file with the start-singleuser.sh command inserted; and then do a helm upgrade using an updated config file with singleuser image?
Hey all - it sounds like there's some useful information in this thread that hasn't made its way into Z2JH yet. Might I suggest that either:
- @cam72cam opens a PR to add a guide for NFS, similar to what's above
- If this isn't a solution we want to "officially" recommend yet for the reasons @yuvipanda mentions above, @cam72cam should write up a little blog post and we can link to this.
What do folks think?
We are currently doing an internal alpha with a setup similar to the one mentioned above and working out any minor issues which come up. @mfox22 I'd be up for either, what do you think?
My biggest concern with how it works at the moment is that a clever user could look at the system mounts and figure out how to do a userspace nfs mount with someone else's directory. I think we could get around that with configuring the PV differently, but I still have a lot to learn in regards to that.
Well if this is "useful functionality that may introduce some bugs because it's in an 'alpha' state" kinda functionality, maybe a blog post kinda thing is better? One reason we added https://zero-to-jupyterhub.readthedocs.io/en/latest/#resources-from-the-community was to make it easier for people to add more knowledge to the internet without needing to be "official" z2jh instructions. :-)
If you like, I'm happy to give a writeup a look-through, and if it ever gets to a point that your team is happy with, we can revisit bringing it into Z2JH?
@yuvipanda I've added instructions for using EFS, Other than the initial setup, it should be pretty similar for a standard NFS server.
Can we consider this issue resolved in this case? Or perhaps we need a small amount of language basically saying what @cam72cam just said?
Maybe we can refactor the initial work I did to have a common PersistentStorage guide with a NFS servewr as the example. Rewrite the EFS page to link to that with some instructions about the initial EFS setup.
Refactoring and clarifying is always welcome @cam72cam - let me know if you'd like a review at some point!
Bumping - still relevant issue, I lack the overview of the quite long discussion though.
I second the bump. I'm having issues understanding how to connect the Hub to NFS.
@vroomanj I still need to break the docs out into 2 different sections. Take a look through https://zero-to-jupyterhub.readthedocs.io/en/stable/amazon/efs_storage.html
Changes for pure NFS setup:
- Skip first step
- Change fs-${EFS_ID}.efs.us-east-1.amazonaws.com to $YOUR_NFS_SERVER
- Change names of efs_ to nfs_
@cam72cam thanks for the work on the docs!
I was getting EFS setup and found what I feel was an easier way and wanted to see what your thoughts were, I was able to remove the uid and fsGid and the chowning by simply performing a chown 1000:100 on the /home directory which I created beforehand on my EFS volume.
Do you see any downsides with this?
