Andrew Gu

icon location New York, NY

Results 159 comments of Andrew Gu

[FSDP2] Used `CommDebugMode` for comm. count test

@pytorchbot merge

[FSDP2] Added test for manual reshard with `reshard_after_forward=False`

@pytorchbot merge

[FSDP2] Added test for manual reshard with `reshard_after_forward=False`

@pytorchbot merge

[FSDP] Warned on wrapping `ModuleList`/`ModuleDict`

@pytorchbot merge

[FSDP] Warned on wrapping `ModuleList`/`ModuleDict`

@pytorchbot merge

[torchtitan][optim] Add fused as an option in train config

I am curious if we have any experiments to see the performance difference with `fused=True`.

[torchtitan][optim] Add fused as an option in train config

@weifengpy `foreach=True` used to be the default, so perhaps your package was before https://github.com/pytorch/torchtitan/pull/386 landed. Without https://github.com/pytorch/torchtitan/pull/386, the optimizer would fall back to `foreach=False` when `fused=False`. 2000 ms for optimizer...

[WIP] full finetune / qlora + ac/offload/optm in bwd

Should we call out that this table assumes that we are only applying QLoRA to the FFNs?

SequenceParallel sharding seems wrong

cc: @tianyu-l maybe? since @wanchaol is out for a bit

SequenceParallel sharding seems wrong

@bosmart I think that if the feed forward has both `w1` and `w3` that are `ColwiseParallel` (like SwiGLU), then we prefer to only redistribute the input from `(S(1),) -> (R,)`...