Custom images launched in SageMaker Studio use Python installation not present in uploaded image

[dburkhardt]

Expected Behavior

1. I attach a custom docker container to my SageMaker Studio session (example Dockerfile)
2. I launch a SageMaker Studio notebook using a Jupyter Kernel from the image
3. The kernel launches the system Python installation at /usr/local/bin referencing packages installed at /usr/local/lib/python3.8/site-packages

Οbserved behavior

The SageMaker Studio notebook launches a kernel session using a python installation not present in the Docker container when uploaded to ECR found at /opt/.sagemakerinternal/conda/bin/python.

This happens with Notebook and Console kernel sessions launched from the image.

Attempted debugging

I can confirm that the Notebook and Console sessions are launching the correct image because I can find my installed packages in /usr/local/lib/python3.8/site-packages from within the Notebook and the Console.

I can confirm this doesn't happen if I launch an Image Terminal from the Studio Session. When I do this, the Docker container is launched as expected and executing $ python points to the expected installation in /usr/local/bin

I think this has something to do with how Jupyter sessions are launched within SageMaker Studio. When I look at the logs on CloudWatch, I see this:

I have a few concerns:

1. Why is SageMaker using conda here when my image doesn't have conda installed? Can I override this? I would have thought the main purpose of Custom images was to be able to use a different version of Python than standard SageMaker Images. Why override this when importing custom images?
2. What does it mean that this warning is getting raised: + echo This is not a DLC/Studio image (found Custom), so not adding Python3 kernel. This should be a SageMaker image. I make it copying the echo_kernel example Dockerfile from this repo. SageMaker has no issues launching the Notebook session from the console.