gpt-fast icon indicating copy to clipboard operation
gpt-fast copied to clipboard

Published 20 hours ago • verified publisher icon pytorch-labs

batching/dynamic batching

Open nivibilla opened this issue 11 months ago • 2 comments

Thanks for the amazing work! It really is super fast at bs=1.

Can batch usecases, or dynamic batching be supported?

nivibilla avatar Feb 27 '24 08:02 nivibilla