OLMo icon indicating copy to clipboard operation
OLMo copied to clipboard

Published 20 hours ago • verified publisher icon allenai

Try PyTorch FSDP "HYBRID_SHARD" strategy

Open epwalsh opened this issue 1 year ago • 0 comments

While this wasn't implemented as of PyTorch 1.13.1, it appears it's going to be in the next release because it's implemented in the master branch: https://github.com/pytorch/pytorch/blob/master/torch/distributed/fsdp/api.py#L31.

epwalsh avatar Mar 13 '23 23:03 epwalsh