TensorRT-LLM
TensorRT-LLM copied to clipboard
NVIDIA
Complex beam search
Hello! Thank you for this incredible project. It has been extremely useful.
I've noticed that while the current beam search method is quite effective, there are some variations like "group beam search" and "diverse beam search" that might provide improved results in some scenarios. Would it be possible to consider supporting these methods in future updates?
"diverse beam search" is supported in TensorRT-LLM, you can enable it through this parameter beam_search_diversity_rate.
I remember "group beam search" is part of "diverse beam search", and we can decide how many groups we want to use. That's what TensorRT-LLM does not support now. The number of group in TensorRT-LLM is fixed when beam_search_diversity_rate is set by non-default value. We will consider how to extend it in the future.
Oh, sorry, I meant to say "constraint beam search" and "group beam search".
We don't support constraint beam search, but we support group beam search now.
Hi folks! Are there updates on constraint beam search? It's something we would like to integrate.
Likewise. Imposing constraints on beam search (like HF's decoding strategies) would be invaluable
Thank you for the request. To prevent discussing many topics into one bug and lead to confusing, please create another feature request bug to request the constraint beam search. Thank you for the help.