mistral.rs
mistral.rs copied to clipboard
Return finish_reason as an Enum - with wrapped stopping word/sequence/tok
I use stop words/sequences to determine next steps after a response. So if the LLM returns stop word A, we perform action X.
I also do the same with other finish reasons - for example retrying with longer context size if the finish_reason is Length.
Current behavior:
- finish_reason is a string
- stop sequences/words are not returned, so knowing which stop sequence was hit is not possible
Proposed behavior
- finish_reason is an enum. this will allow for easier matching
- the finish_reason stop sequence/word/tok variant includes the string that caused the end of inference