DeepSpeed-MII
DeepSpeed-MII copied to clipboard
Provide async api in MII client
Currently the only public method available to generate
is blocking event loop. It would be great to provide async version of generate method.
Hi @ttim we can add this! Before I begin work, could you share how you plan to use an async generate
API? This will help ensure that we create a solution that works well for your needs. Thanks!
@mrwyattii something like current generate
method but async
would be enough. I would like to call this method from another method which is declared async
itself (with a possibility of multiple runs happening at the same time). Currently I spin up new thread, create event loop, create MII client and query from it which seems suboptimal.