TensorRT-LLM icon indicating copy to clipboard operation
TensorRT-LLM copied to clipboard

[Feature request] Add LogitsProcessor class support in C++ Executor API

Open chiendb97 opened this issue 8 months ago • 0 comments

Hi team, I would like to use the LogitsPostProcessor in the C++ Executor API to control the generation of language models. However, unlike frameworks like Hugging Face, vLLM, or the implementation in Model Runner, which support class-based approaches, this feature currently only supports functions. This limitation makes implementation challenging. Could the TensorRT-LLM team consider adding support for this feature in TensorRT-LLM? Thank you.

chiendb97 avatar May 27 '24 04:05 chiendb97