tensorrtllm_backend icon indicating copy to clipboard operation
tensorrtllm_backend copied to clipboard

gpt_model_path with Triton's S3 based model repository support

Open sacdroid opened this issue 1 year ago • 3 comments

I am trying externalize model artifacts to S3 using Triton's Cloud Storage support for model repository. I am able to get this working for pre/postprocessing tokenizer model instances using

parameters {
  key: "tokenizer_dir"
  value: {
    string_value: "$$TRITON_MODEL_DIRECTORY/1"
  }
}

and the replacing this in model.py.

How can I achieve the same gpt_model_path ? I looked at the code and it does not seem to support dynamic path today. Do you have any alternatives which does not require me to include model artifacts in docker container or have external mount?

sacdroid avatar Dec 02 '23 04:12 sacdroid