Devin Chotzen-Hartzell

icon indicating a website link https://www.linkedin.com/in/devin-chotzen-hartzell-aaab371b5/ icon indicating an email [email protected]

icon company Codeium icon location training infrastructure @ codeium

Results 2 comments of Devin Chotzen-Hartzell

[BUG] Checkpoint saving is slow for zarr backend + distributed optimizer

yes I've tried it, we ran into some other issue regarding saving the optimizer step with dp_partitions >= 2. will file another bug for that when I have a chance...

[BUG] Checkpoint saving is slow for zarr backend + distributed optimizer

Hi @deepakn94, which kinds of checkpoint resharding are meant to be supported for the torch_dist backend? I'm unable to load a (D, P, T) = (2, 2, 2) checkpoint into...