Implement end_sequence for text generation

[juliensalinas]

It would be awesome if your generate_batch() function could support an end_sequence parameter (stopping the generation as soon as a specific token is reached): https://huggingface.co/docs/transformers/v4.21.1/en/main_classes/text_generation#transformers.generation_utils.GenerationMixin

In my opinion this parameter is crucial as soon as one wants to use few-shot learning with these generative models (https://nlpcloud.com/effectively-using-gpt-j-gpt-neo-gpt-3-alternatives-few-shot-learning.html).

Thanks a lot!

[guillaumekln]

This should be easy to add.

Do you think this feature is useful right now even though GPT-J and GPT Neo are not yet supported?

[NeonBohdan]

Hi @guillaumekln is it possible to implement EOS sequence of tokens For example this model don't use a token to stop generation but a sequence ["<", "bot", ">:"]

[guillaumekln]

For this example, it seems that the token ">:" is already quite specific and can be used as the end token. Did you try that?

[NeonBohdan]

@guillaumekln Thanks Looking at the problem from a development time point of view, it's easier to use the "<" or ">:" token. You are right, I just notify you that such models already exist(and new may come)