How is GptManager used in Triton backend?

[ekagra-ranjan]

I see that Triton backend creates an object of GptManager which gets passed the engine dir. However, I unable to see any code that shows how this GptManager is being called. All I can see is backend calling some Triton function but the GptManager is not a function arg to those calls so I am curious how is the engine being called from Triton backend.

Can I please get some pointers to the code which does this?

Thanks!

[byshiue]

gptManager is defined in this header file https://github.com/triton-inference-server/tensorrtllm_backend/blob/bf5e9007a3f16c7fc76cb156a3362d1caae198dd/inflight_batcher_llm/src/model_instance_state.h#L39, but the implementation is not opened.