text-generation-inference

text-generation-inference copied to clipboard

HF has contrastive sampling, ideally we could use that to sample here too.

Contrastive sampling is very expensive to compute and would require a large change in the codebase to be supported. If there is enough interest we can add it but for now it is not a high priority.

Well at the same time I added best_of which is even more expensive...

haha, no worries :) we saw some good results with contrastive. but not urgent at all. thanks for the consideration