Does python model support venv-pack for Creating Custom Execution Environments

[marcozzxx810]

Is your feature request related to a problem? Please describe. Currently, the Python model document only mentions it support conda-pack.

Describe the solution you'd like It will be great if it supports venv-pack and provide guide line about how to set it up

[indrajit96]

CC @nv-kmcgill53 @mc-nv

[marcozzxx810]

any update? @indrajit96

This is venv-pack: https://github.com/jcrist/venv-pack/

[shireen-bean]

Is your feature request related to a problem? Please describe. Currently, the Python model document only mentions it support conda-pack.

Describe the solution you'd like It will be great if it supports venv-pack and provide guide line about how to set it up

Yes, you can use venv-pack as well as a vanilla useage of tar-ing your virtual environment for use in building a custom execution environment with the caveat of you will need to make available the python interpreter if you are using a python version other than the interpreter version included in the tritonserver image. Right now the current python included is python 3.10 so I've needed to compile the stubs and build custom environments for my version 3.9

With virtualenvs as opposed to conda envs, the python interpreter is symlinked so you might find that when you venv-pack or tar your virtualenv, you get errors indicative of not being able to find your python interpreter. I've had these surface like:

Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Consider setting $PYTHONHOME to <prefix>[:<exec_prefix>]
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = '/usr/local/lib/python3.9'
  program name = 'python3'
  isolated = 0
  environment = 1
  user site = 1
  import site = 1
  sys._base_executable = '/usr/bin/python3'
  sys.base_prefix = '/usr/local'
  sys.base_exec_prefix = '/usr/local'
  sys.platlibdir = 'lib'
  sys.executable = '/usr/bin/python3'
  sys.prefix = '/usr/local'
  sys.exec_prefix = '/usr/local'
  sys.path = [
    '/usr/local/lib/python3.9',
    '/usr/local/lib/python39.zip',
    '/usr/local/lib/python3.9',
    '/usr/local/lib/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

and

/usr/app/model-repository/model/triton_python_backend_stub: error while loading shared libraries: libpython3.9.so.1.0: cannot open shared object file: No such file or directory

I was able to passed those errors by including the python interpreter in the tritonserver image used for serving.

Example

An example of how to do this:

apt install python3.9-dev python3.9-venv python3.9-full -y

virtualenv -p python3.9 venv
source venv/bin/activate

pip install -r custom_requirements_file.txt

echo "Compiling Triton Python backend stub process from source."

git clone \
  --depth 1 \
  --branch r${triton_version} \
  https://github.com/triton-inference-server/python_backend

# Call cmake with the appropriate flags for you use case
cmake -DTRITON_ENABLE_GPU=ON -DTRITON_BACKEND_REPO_TAG=r24.03 -DTRITON_COMMON_REPO_TAG=r24.0 ...
# Call make to build the stub process
make triton-python-backend-stub

deactivate

# I use tar but you could also use venv-pack here
tar -czf venv.tar.gz -C venv .

# Clean up build artifacts
...
# a conda-env would allow you to copy over just the venv.tar.gz file and the python stub
# but for tar or venv-pack you need to maintain the python interpreter for non-python310```