using extraFiles makes the parent directory read-only, which can often break JupyterHub

[cboettig]

Bug description

It seems like a common case for singleluser.extraFiles would be to load configuration files for JupyterHub and related extensions. For instance, I'd like to use this to provide a pre-configured config.json for the jupyter-ai extension, which lives in $XDG_HOME/jupyter/jupyter_ai .

However, setting this with something like:

singleuser:
  extraFiles:
    jupyter_ai_config.json:
      mountPath: "/opt/share/jupyter/jupyter_ai/config.json"
      data: 
        model_provider_id: "openai-chat-custom:llama3"

makes the hub fail to launch any image, due to a permissions error introduced on jupyter/runtime, which is found in the parent folder. This seems to be a known issue, (though I didn't see this side-effect documented in the docs).

(Maybe this counts as a feature request rather than a bug, but because a seemingly valid use of the documented option for extraFiles leads to the hub not even launching it feels more like a bug).

How to reproduce

add config listed above to jupyterhub values.yaml and deploy

Expected behaviour

jupyter-ai extension, if installed on the image, has the provided config. Ideally the file itself is read-write permissions to the jupyter user, though read-only would be okay to.

Actual behaviour

JupyterHub fails to launch pod, kubectl logs show pod gets stuck on a permission error on jupyter/runtime

Your personal set up

https://github.com/boettiger-lab/k8s/blob/main/jupyterhub/thelio-config.yaml

[consideRatio]

Does the ~/.jupyter directory already exist in the container image with the correct permissions before you mount the extra file?

@manics asked an excellent question. I bet the permission issues arise because folders created in order to mount the file will be created with too low permissions by default. If they were already created ahead of time with the right permissions, there wouldnt be an issue.

[cboettig]

@consideRatio ah brilliant, I missed that, yes that resolves the immediate issue. (slightly not ideal if a user brings their own image, which we do a lot via jupyter fancy profiles, but we can deal).

Unfortunately the config file itself is still read-only, which at least in the case of jupyter-ai breaks things:

jupyter_ai | extension failed loading with message: OSError(30, 'Read-only file system')

is it possible to avoid / work around this?

(aside, I see we can set mode , e.g. to 0666, but since the file is mounted read-only this doesn't actually grant write access).

[manics]

extraFiles are implemented as Kubernetes Secrets which are mounted into the pod as a file https://github.com/jupyterhub/zero-to-jupyterhub-k8s/blob/6cadee8060c55692b3d831d1d10ab06e04e418e2/jupyterhub/templates/singleuser/secret.yaml so there's no way to make the file writeable, that'd be equivalent to giving users write access to the Kubernetes API.

If you want a writeable file I'd use a postStart hook instead https://z2jh.jupyter.org/en/stable/jupyterhub/customizing/user-environment.html#about-user-storage-and-adding-files-to-it

[cboettig]

@manics ok cool, that makes perfect sense. postStart seems the way to go. is echo 'all-my-file-contents-on-one-line' > /to/path the way to go there or are there other options than exec.command?

[manics]

postStart is provided by Kubernetes: https://kubernetes.io/docs/tasks/configure-pod-container/attach-handler-lifecycle-event/ so it can be shell command, either with the content of the file in-line or it could pull the file from an external resource such as git.

Another option if you're using a jupyter/docker-stacks image is to use a startup hook https://jupyter-docker-stacks.readthedocs.io/en/latest/using/common.html#startup-hooks