vllm_backend icon indicating copy to clipboard operation
vllm_backend copied to clipboard

Add resolve_model_relative_to_config_file config option

Open Legion2 opened this issue 5 months ago • 4 comments

Currently relative paths to local models are resolved relative to the triton server process. However when deploying models to a central model registry one may not know in advance where the model files will end up in the local file system of an triton inference server container. Therefore relative paths should be used to point from the model.json to the directory containing the actual model files. The relative paths are maintained when deploying models and downloading them from a registry. To make this work relative paths in model.json must be resolved relative to the model.json file location.

This PR adds a new property (resolve_model_relative_to_config_file) to the model.json which allows to enable this relative resolution behavior. By default the old behavior is maintained.

I added test for the new feature, but do not understand how to execute them and where they are executed in ci.

Legion2 avatar Jan 22 '24 11:01 Legion2