DeepSpeed-MII
DeepSpeed-MII copied to clipboard
Streaming api seems broken
Currently while using streaming API output seems off - there is no spaces in response and it's unclear where to add them.
I've tried printing in streaming_fn
following strings:
-
"".join(item.response)
in this case there is no spaces in output -
" ".join(item.response)
in this case still no spaces -
"".join(item.response) + " "
- too many spaces, seems like after each token
I'm not sure if it's something wrong I do or something is broken, any help would be great!
I have the same issue
Hi @ttim and @PawanOsman we have not officially released the streaming API for this reason. @jeffra is actively working to fix these problems and you can expect an update and official release of this feature very soon!