DeepSpeed icon indicating copy to clipboard operation
DeepSpeed copied to clipboard

Published 20 hours ago • verified publisher icon microsoft

Add support for T5 for deepspeed-inference

Open HeyangQin opened this issue 2 years ago • 2 comments

This is the PR to add support to T5. Currently it is still work in progress. A lot of the codes are adapted from https://github.com/microsoft/DeepSpeed/pull/2451

HeyangQin avatar Feb 16 '23 18:02 HeyangQin

Hi @HeyangQin, thanks for your great work! Is there any update on the timeline of merging T5 support?

alexcoca avatar Mar 01 '23 19:03 alexcoca

For whatever reason, it seems it is requiring a lot more VRAM when this policy is injected.

baptistejamin avatar May 02 '23 18:05 baptistejamin

@HeyangQin @loadams I take it that this is not on your roadmap?

alexcoca avatar Sep 07 '23 22:09 alexcoca

Hello @alexcoca. T5 support should already be live with https://github.com/microsoft/DeepSpeed/pull/2962. Please feel free to let us know if you need more information

HeyangQin avatar Sep 07 '23 22:09 HeyangQin