Jeff Rasley

I've repro'd your issue, will let you know when i have a fix. Our support on windows is unfortunately not as thoroughly tests as on linux. I recognize how funny...

I repro'd this on a windows box that does not have a GPU. Can you confirm that torch sees your gpu from windows? Can you share the results of `torch.cuda.is_available()`...

This is definitely on our upcoming TODO list to investigate. Are you saying you've tried your own custom kernel injection policy and it (partially?) works?

Thank you for reporting this! I've verified we can repro this on our side as well, but only when using >1 gpus. There's a gap currently in our CI tests...

Hi @lanking520, I just tried all your repro steps above and was not able to repro the stack trace. Can you confirm what `transformers` version you are using? I tried...

Also, just to double check, you can run fine if you remove `deepspeed.init_inference` right?

We’ve definitely been watching if/when pytorch will support M1, it sounds like it’s planned though. https://github.com/pytorch/pytorch/issues/47702 Specifically see this comment: https://github.com/pytorch/pytorch/issues/47702#issuecomment-965625139 In terms of DeepSpeed support for M1 I suspect...

I should also note that I think we have similar complexities with `communication_data_type`, which we only support in some configs but not sure if we error out explicitly on cases...

@joehoover, can you give it a try now with this PR linked above? I think we have this fixed now.

Hi @TianhaoFu can you share `ds_report` with me? I am curious on what deepspeed version or commit hash you were on. I am trying to reproduce your issue. Also if...