Karan Dhiman comments

Results 5 comments of


                                            Karan Dhiman

[BUG] Inconsistent loss between `overlap_comm=true` and `overlap_comm=false`

@clumsy

[BUG] Inconsistent loss between `overlap_comm=true` and `overlap_comm=false`

@dashstander - thanks for the response. Unfortunately, I haven't had time to dive into this yet. If I have any updates, I'll report them here.

RuntimeError: No such operator tutel_ops::cumsum

I get this error after setting `FAST_CUMSUM=0` ```bash [1,319]:│ /root/.local/lib/python3.9/site-packages/tutel/impls/fast_dispatch.py:108 in │ [1,319]:│ update │ [1,319]:│ │ [1,319]:│ 105 │ │ if self.is_cuda != indices_[0].is_cuda: │ [1,319]:│ 106 │ │...

RuntimeError: No such operator tutel_ops::cumsum

> Gotcha, this problem is not from `tutel::cumsum`. Instead, you may perform an improper installation of Tutel that only enables CPU support rather than CUDA. > > The root cause...

RuntimeError: No such operator tutel_ops::cumsum

Thanks for the suggestion! I'm using an older driver && once I added nv repo, I was able to select the version that I needed. `add-apt-repository "deb https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2004/x86_64/ /"`.