mistral.rs icon indicating copy to clipboard operation
mistral.rs copied to clipboard

Return finish_reason as an Enum - with wrapped stopping word/sequence/tok

Open ShelbyJenkins opened this issue 4 months ago • 0 comments

I use stop words/sequences to determine next steps after a response. So if the LLM returns stop word A, we perform action X.

I also do the same with other finish reasons - for example retrying with longer context size if the finish_reason is Length.

Current behavior:

  • finish_reason is a string
  • stop sequences/words are not returned, so knowing which stop sequence was hit is not possible

Proposed behavior

  • finish_reason is an enum. this will allow for easier matching
  • the finish_reason stop sequence/word/tok variant includes the string that caused the end of inference

ShelbyJenkins avatar Sep 27 '24 17:09 ShelbyJenkins