Pip install library to user from Jupyter Notebook cell does not work on K8s environment

[lresende]

Add the following in your notebook !pip install --user --no-cache opencv-python and import cv2

After the import is executed, you will see an error related to ImportError Traceback (most recent call last) /usr/local/bin/kernel-launchers/python/scripts/launch_ipykernel.py in <module> ----> 1 import cv2

Full details on the image below:

[esevan]

In my case this issue is raised by two reasons.

1. missing OS packages. Needed to install those by !apt-get install libsm6 libxrender1 libfontconfig1.

opencv-python needs OS package such as libsm6 libxrender1, and libfontconfig1.

This issue was resolved by latest python3 kernel since it's built on minimal-notebook including those packages.

2. sys.path is not up-to-date. I don't know much about detail, but opencv-python installs cv2 package somewhere not in initial sys.path. After restarting kernel, it worked.

[lresende]

Thanks @esevan , I am finding exactly the same issues as you. if you have a working solution, do you want to provide a patch via GitHub PR ?

[esevan]

@lresende Glad to hear that. AFAIK, First case is already resolved in kernel-py and kernel-tf-py, but not in kernel-tf-gpu-py. https://github.com/jupyter/enterprise_gateway/blob/9c38f1e14917173bcf8e7f204a5f3a97f7e89a31/etc/docker/kernel-tf-gpu-py/Dockerfile#L2 I'm going to add commands to Dockerfile for installation packages required.

I have no other solution than restarting kernel for the second case, but let me take a look at it more.

I can send PR by 12th, Apr because there're several pending issues I have to resolve.
If it's urgent then let another person do this work.

[kevin-bates]

Got a couple comments.

1. @esevan - thank you for the update - I don't think this is time-sensitive. When you do make the changes to the Dockerfile for the gpu image, could you also make the same changes to the docs. Although we no longer publish the gpu image (there are some licensing concerns relative to CUDA that we're looking into), we do use this as an example for how to build a custom kernel image that is not built from a jupyter docker-stacks image. So this would be a great example of the kinds of things people would need to deal with.
2. I think we're going to need to change the permissions on /home/jovyan (and below) in order to properly support user's wishing to run their pods as a different UID (including pip install --user ...). This won't affect the HOME setting, but it does affect the effective UID. As a result, I believe we should make all directories executable and all files writable for others. I believe a statement like the following would be sufficient: chmod -R go+wX /home/jovyan. I go ahead and include the g for groups since there may be files currently set at rw-r--r-- and without it, we'd have rw-r--rw- which is odd.

Thoughts on item 2?

[esevan]

@kevin-bates My opinion about item 2.

1. If we have to set 1000/100 uid to /home/jovyan, then your saying is right. However I don't prefer to change permission in Dockerfile because it's static change, but uid is dynamic value in EG design. Can we change permission of /home/jovyan in initContainer if it's in need? Refer the stack overflow answer.

2. When volumeMount for /home/jovyan is specified, /home/jovyan is not a docker directory and it's not affected by chmod or chown in Dockerfile. In this case we don't need to set any permission of /home/jovyan in docker. It means no need to change current design. Letting NFS management system take in charge of management of directory permission is reasonable.

[kevin-bates]

@esevan - I agree with your points - especially the notion that those that will change the UID will likely bring their own home directory via mounts. If they don't, they'll quickly find they need to. So, in essence, use of runAsUser will imply mounting over /home/jovyan.

Regarding init containers, I'm not that familiar with them. Can they access the contents of the app (kernel) container (e.g., /home/jovyan)? Since they run before the app container, I don't see how they would be applicable. Seems like they're useful for starting or ensuring external resources are present and available. That said, we don't prevent users from configuring init containers. The other downside to init containers is that they will increase the overall kernel startup time - which we try to minimize.

I'm okay with leaving things as is. Although unconventional, I really don't see the harm in making a home directory writable by all within a container. Is there something I'm missing? Just curious.

[esevan]

Okay, I maybe misunderstood your point. I thought runAsUser implicitly says "I'm running a pod on NFS volume!".

Unless NFS volume is mounted, initContainer wouldn't affect home directory.

In the case where only UID is set (no volume mounted), your approach is great to support the user is able to write and execute.

Thanks! I will add that too.

[kevin-bates]

@esevan - Let's not change the perms in the image for now. I think it's appropriate to assume that if runAsUser is used, that usage would also include a mounted home directory that would contain the appropriate perms corresponding to the target user. Should we find otherwise, we can cross that bridge then. Does that sound okay to you?

[esevan]

I didn't address sys.path issue but missing OS dependency issue. I've checked the fact that sys.path doesn't include user site packages before first user package is installed. It is about neither opencv-python nor EG.

[lresende]

See comment above, we still need to address the sys.path (e.g. PYTHONPATH) configuration and add user installation folder or make global one writable by jovyan.

[kevin-bates]

@lresende - do you have any update on this?