DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

how to set stop tokens?

Open PawanOsman opened this issue 7 months ago • 3 comments

Is there a way to implement a stopping_criteria in DeepSpeed MII for text generation, to stop at a newline or custom token?

PawanOsman avatar Nov 13 '23 18:11 PawanOsman

Hi @PawanOsman! I added this feature in #280, but I was observing unexpected behavior in some cases so I did not add to our official documentation. I plan to debug the problem I was seeing and update the docs when I'm sure it's working correctly this week.

In the meantime, you can test it yourself and report any issues that arise:

client.generate(prompt, stop="\n") (The input can either be a token_id or string)

mrwyattii avatar Nov 13 '23 19:11 mrwyattii

Thank you so much, is it possible to give it a list of tokens? or it can be only a single token?

PawanOsman avatar Nov 13 '23 20:11 PawanOsman

Currently we support just a single token, but I can expand this feature to support multiple.

mrwyattii avatar Nov 13 '23 21:11 mrwyattii