zejunchen-zejun comments

Results 16 comments of


                                            zejunchen-zejun

[BUG] Build cutlass with arch=100a failed

It works! Thank you!

[collective op][cuda graph] capture collective ops but got an HIP error: operation not permitted when stream is capturing

Hi, @wenkaidu @gilbertlee-amd Could you help take a look? Thank you

[collective op][cuda graph] capture collective ops but got an HIP error: operation not permitted when stream is capturing

Hi, @thananon Thank you for help! `we have encountered this issue before and this is due to ROCm 7.0 behavior change to match CUDA. We no longer allow certain operations...

[collective op][cuda graph] capture collective ops but got an HIP error: operation not permitted when stream is capturing

Hi, @amd-nicknick Thank you for help. We will run the reproducer on B200 machine and check the behavior.

[collective op][cuda graph] capture collective ops but got an HIP error: operation not permitted when stream is capturing

Hi, @amd-nicknick @thananon I tested the reproducer on B200 and the behavior is `torch.dist.all_reduce cannot be captured by cuda graph on NV platform`. So I think the ROCm's behavior does...

[collective op][cuda graph] capture collective ops but got an HIP error: operation not permitted when stream is capturing

Hi, @amd-nicknick @thananon Thank you for help. You are right. It makes perfect sense. The first torch.dist op will trigger the lazy initialization of RCCL, which calls the hipFree internally....