Why should I run custom images as sagemaker-user?

[peter-wimsey]

The examples in this repo and the image configuration in SageMaker Studio assume that custom images are run as user sagemaker-user with UID 1000 and GUID 100.

However, at least some of SageMaker's own images (Data Science and PyTorch for example, have not tested all) run as root.

Running as something other than root makes installing into the kernel image at runtime difficult:

$ pip install some-package
Defaulting to user installation because normal site-packages is not writeable
...

and I end up with files in the user's home directory where they don't get cleaned up.

Why should I not just run as root like SageMaker's built-in images?