Wanchao

Results 84 comments of Wanchao

Allow initialization of optionals from convertible type U (or optional<U>)

Hi @akrzemi1, any updates on this one? I am seeing https://en.cppreference.com/w/cpp/utility/optional/optional, the library is missing constructor 8. Do you have any plan on adding it?

[WIP] Make allreduce compatible with fx ProxyTensor

> Thanks! Let me try that. feel free to land this first, we can do this in follow up PRs

[Feature] Plan to add `model_register`

@XinDongol I think that's a good idea, the way this could work is that `register_model` can insert new entries into various module's module level dict (i.e. [this](https://github.com/pytorch/torchtitan/blob/main/torchtitan/models/__init__.py#L9)), there are couple...

[dtensor] delete the old unused mesh_alltoall

@pytorchbot merge

[dtensor] delete the old unused mesh_alltoall

@pytorchbot merge

[dtensor] delete the old unused mesh_alltoall

@pytorchbot merge -f "failures not related"

Checkpoint saves failing for eager mode training

@wz337 mentioned that she tried it today and it works on her end. @wz337 could you work with @chauhang to resolve this?

Checkpoint saves failing for eager mode training

I just tried this locally on the debug_model.toml, and it also seems working on my end 🤔 could there be setup differences?

Question: tp able to run a model which not able to fit a single batch on GPU?

@lucasjinreal As @awgu mentioned, any form of parallelisms or AC might help reduce the peak GPU memory usage. If you have applied some of the techniques (i.e. AC or reduced...

torch.compile each TransformerBlock instead of the whole model

> Should we be able to close #61 after this PR? > > Also, do we need to run end-to-end numerics testing? @awgu yeah I think it should resolve that,...