Andrew Gu

icon location New York, NY

Results 159 comments of Andrew Gu

Question on Model Init

I would really appreciate some pointers to the complicated initialization to learn more about it. And yes, I think that the seed checkpoint can be used to avoid the meta...

Question on Model Init

cc: @wanchaol @tianyu-l The above two pointers are good examples of real-model init methods that do not fit our current meta-device init flow. As far as I can tell, both...

Update requirements.txt

@qiziAI Thanks for the PR! Could you provide some more details of the conflict for our understanding?

Modify FLOPs in MFU calculation for casual mask when using FlashAttention.

There was some past discussion on this (https://github.com/pytorch/torchtitan/pull/280).

[FSDP2] Fixed 2D clip grad norm test

Failure are all inductor-related, not FSDP2-related.

[FSDP2] Fixed 2D clip grad norm test

@pytorchbot merge -i

[FSDP2] Fixed 2D clip grad norm test

@pytorchbot rebase -s

[FSDP2] Fixed 2D clip grad norm test

@pytorchbot merge

[FSDP2] Fixed 2D clip grad norm test

@pytorchbot rebase -s

[FSDP2] Fixed 2D clip grad norm test

@pytorchbot merge