DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Streaming api seems broken

Open ttim opened this issue 7 months ago • 2 comments

Currently while using streaming API output seems off - there is no spaces in response and it's unclear where to add them. I've tried printing in streaming_fn following strings:

  • "".join(item.response) in this case there is no spaces in output
  • " ".join(item.response) in this case still no spaces
  • "".join(item.response) + " " - too many spaces, seems like after each token

I'm not sure if it's something wrong I do or something is broken, any help would be great!

ttim avatar Nov 10 '23 23:11 ttim

I have the same issue

PawanOsman avatar Nov 13 '23 11:11 PawanOsman

Hi @ttim and @PawanOsman we have not officially released the streaming API for this reason. @jeffra is actively working to fix these problems and you can expect an update and official release of this feature very soon!

mrwyattii avatar Nov 13 '23 16:11 mrwyattii