pytorch-image-models icon indicating copy to clipboard operation
pytorch-image-models copied to clipboard
verified publisher icon huggingface

[FEATURE] DeepSpeed/ZeRO Support

Open rwightman opened this issue 4 years ago • 0 comments

Along with updated training/validation components in #458 for TPU support, support use of DeepSpeed/ZeRO

  • https://pytorch.org/docs/master/distributed.optim.html#torch.distributed.optim.ZeroRedundancyOptimizer
  • https://github.com/microsoft/DeepSpeed

It would be fairly easy to support w/ current training code, however the cahnges in #458 are going to be so significant that I do not want to continue two branches of code.

rwightman avatar Mar 11 '21 21:03 rwightman