vllm_backend
vllm_backend copied to clipboard
Add resolve_model_relative_to_config_file config option
Currently relative paths to local models are resolved relative to the triton server process. However when deploying models to a central model registry one may not know in advance where the model files will end up in the local file system of an triton inference server container. Therefore relative paths should be used to point from the model.json
to the directory containing the actual model files. The relative paths are maintained when deploying models and downloading them from a registry. To make this work relative paths in model.json
must be resolved relative to the model.json
file location.
This PR adds a new property (resolve_model_relative_to_config_file
) to the model.json
which allows to enable this relative resolution behavior. By default the old behavior is maintained.
I added test for the new feature, but do not understand how to execute them and where they are executed in ci.