Ang Wang
Ang Wang
@ymwangg thanks for your reply. yeah, we use three_fry generator as default for now. Hope that one day we could use the curand RNG wihout memory problems. ^_^
set three_fry as the default RNG for GPU.
Okay, I'll take a look at these ops.
pai-torchacc and torchacc are the same thing, developed by the PAI team at Alibaba Cloud. You can try it out by visiting [flashmodels](https://github.com/AlibabaPAI/flashmodels) or review the documentation for [torchacc](https://torchacc.readthedocs.io/en/latest/).
Which document are you referring to? xlarun is now deprecated. You can use torchrun directly, and take a look at the FSDP example: https://torchacc.readthedocs.io/en/latest/dist/fsdp.html#fsdp
We have not tested torchacc with CTR models before, but you can try it by wrapping your self.model with torchacc.accelerate. This document might be helpful to you: https://torchacc.readthedocs.io/en/latest/dist/dp.html.