Takuya Kato issues

Repositories
Issues
Comments

Results 1 issues of


                                            Takuya Kato

[BUG] vp>1 and TORCH_NCCL_AVOID_RECORD_STREAMS=1 leads to incorrect loss behavior

**Describe the bug** The loss curve of a training run with the following configuration - virtual-pipeline-parallel-size > 1 - TORCH_NCCL_AVOID_RECORD_STREAMS=1 does not match with the curves of training runs with...