Alp Dener comments

Repositories
Issues
Comments

Results 42 comments of


                                            Alp Dener

create_communicator_grouped2 may trigger uninit value memory issue(randomly crash) when you train more iterations.

@anderson101866 This should be fixed now in TE/main as of PR #1087. Could you check and close the issue if resolved?

FSDP: How to do all-gather using FP8?

Hi @vgoklani -- TE modules can be initialized under the `with te.fp8_model_init():` context to allocate their primary weights in FP8 (as `te.Float8Tensor`s) instead of allocating at a higher precision and...