DeepSpeed-MII icon indicating copy to clipboard operation
DeepSpeed-MII copied to clipboard

Provide async api in MII client

Open ttim opened this issue 1 year ago • 2 comments

Currently the only public method available to generate is blocking event loop. It would be great to provide async version of generate method.

ttim avatar Nov 06 '23 21:11 ttim

Hi @ttim we can add this! Before I begin work, could you share how you plan to use an async generate API? This will help ensure that we create a solution that works well for your needs. Thanks!

mrwyattii avatar Nov 09 '23 00:11 mrwyattii

@mrwyattii something like current generate method but async would be enough. I would like to call this method from another method which is declared async itself (with a possibility of multiple runs happening at the same time). Currently I spin up new thread, create event loop, create MII client and query from it which seems suboptimal.

ttim avatar Nov 10 '23 23:11 ttim