Joe Cummings

icon company @pytorch @facebookresearch icon location nyc

Results 278 comments of Joe Cummings

add pretraining code and multi modal support?

Closing b/c no activity in awhile and we have added multimodal support.

[bug] regression tests failing on nightlies

This has been fixed.

Investigate PreferenceDataset truncation

@SalmanMohammadi Can this be closed?

Explain magic numbers in our tests

Shhhhh it's a secret

[FEATURE REQUEST] Compile model+loss_fn together

cc @msaroufim

[FEATURE REQUEST] Compile model+loss_fn together

Heads up @felipemello1, Evan merged the FSDP2 changes into the full_finetune_distributed recipe so that is now officially FSDP2, not FSDP1. And for FSDP2, we've seen a few issues with compile....

[FEATURE REQUEST] Compile model+loss_fn together

We now compile the loss, as well.

Very High Ram Usage when resuming training from unsharded recipe_state

I think adding support for Distributed Checkpointer would address this - can someone from DCP confirm? @LucasLLC @pbontrager DCP is definitely on our radar as something we want to integrate....

[debug] compile FSDP2 recipe

bump

[debug] compile FSDP2 recipe

@msaroufim Is there a method on NF4Tensor that could be easily implemented to enable FSDP2 + QLoRA + Compile?