Rui Wang

Fatal Python error: Aborted

1

Hi, I was trying to test `https://github.com/NVIDIA/apex/tree/master/apex/contrib/openfold_triton` with triton but encountered this error and cannot find the solution anywhere. It'd be great if I could get some pointers to check...

Installation failed with cmake error

21

Hi, We are testing our new Hopper machines (H800/H100) and trying to use fp8 for training for the first time, but are having trouble installing `TransformerEngine`. It reports ` RuntimeError:...

Massively reduce LayerNorm/RMSNorm training memory usage by sharing saved tensor with other parts of the networks

1

Saving the output of the normalization instead of the input reduces the memory cost in modern networks, where the output is going to be saved anyways (e.g., a Linear layer)...