text-generation-inference

text-generation-inference copied to clipboard

Are there any plans to support logit processors via an additional API parameter? For instance, the OpenAI API provides a logit_bias parameter, that is applied to the token distribution during generation.

This would be great for http://github.com/eth-sri/lmql, where we heavily rely on token masking to enforce constraints during decoding.

Thank you for your hard work on this, it is an amazing project for the general inference space.

Also very interested in this. For tasks like classification, it can be really useful to constrain the output to just some select logits. Especially when you want to have a "confidence" calculation

🤗 transformers just landed the SequenceBiasLogitsProcessor in their main branch, meaning this should be fairly straightforward to also implement here now as well!

Hi, am interested in this as well. Any chance of this being implemented?