tensorrtllm_backend
tensorrtllm_backend copied to clipboard
gpt_model_path with Triton's S3 based model repository support
I am trying externalize model artifacts to S3 using Triton's Cloud Storage support for model repository. I am able to get this working for pre/postprocessing tokenizer model instances using
parameters {
key: "tokenizer_dir"
value: {
string_value: "$$TRITON_MODEL_DIRECTORY/1"
}
}
and the replacing this in model.py.
How can I achieve the same gpt_model_path ? I looked at the code and it does not seem to support dynamic path today. Do you have any alternatives which does not require me to include model artifacts in docker container or have external mount?