llama.cpp icon indicating copy to clipboard operation
llama.cpp copied to clipboard

Add parameter to ignore end of text token

Open slaren opened this issue 1 year ago • 0 comments

Adds the --ignore-eos switch which prevents generation of the end of text (eos) token. This can be useful to avoid unexpected terminations in interactive mode and to force the model to generate longer output.

This is implemented by setting the logits of the eos token to zero, which seems to work well enough, but I am not sure if there may be any unwanted side effects.

slaren avatar Mar 15 '23 21:03 slaren