NathanGodey

Release the intermediate checkpoints?

2

Hello! Thank you for the great work. I was wondering if you planned to release the intermediate checkpoints for all pretrained models as in Pythia (https://arxiv.org/pdf/2304.01373.pdf)?

`strict = False` does not work when the checkpoint is distributed

1

### Bug description When loading a sharded checkpoint with: ```python fabric.load(ckpt_path, state, strict = False) ``` the `_distributed_checkpoint_load` function called in the `FSDPStrategy` will raise an error if a checkpoint...