deepbind_train_dream5.py fails on test step due to CUDA error in Memcpy2DAsync

[sthodup1]

This python fails with the aforementioned CUDA error. The error looks like:

Predicting... ['TF_34']

AssertionError: CUDA error in Memcpy2DAsync: initialization error. copy_from(...) raised an exception AssertionError: CUDA error in Memcpy2DAsync: initialization error. in smat_cuda/copy.cu:190 Traceback (most recent call last): File "libs/kangaroo/kangaroo/predict.py", line 405, in call model = load_model(modelinfo) File "libs/kangaroo/kangaroo/predict.py", line 107, in load_model model = cp.load(f) File "libs/smat/py/smat/smat.py", line 230, in setstate dll.api_copy_from(self._ptr,data.ctypes.data_as(c_void_p),data.strides[0],data.strides[1]) # Copy from data File "libs/smat/py/smat/smat_dll.py", line 183, in call raise SmatException(msg) SmatException: copy_from(...) raised an exception AssertionError: CUDA error in Memcpy2DAsync: initialization error. in smat_cuda/copy.cu:190

Error in Worker... copy_from(...) raised an exception AssertionError: CUDA error in Memcpy2DAsync: initialization error. in smat_cuda/copy.cu:190 Traceback (most recent call last): File "libs/kangaroo/kangaroo/predict.py", line 405, in call model = load_model(modelinfo) File "libs/kangaroo/kangaroo/predict.py", line 107, in load_model model = cp.load(f) File "libs/smat/py/smat/smat.py", line 230, in setstate dll.api_copy_from(self._ptr,data.ctypes.data_as(c_void_p),data.strides[0],data.strides[1]) # Copy from data File "libs/smat/py/smat/smat_dll.py", line 183, in call raise SmatException(msg) SmatException: copy_from(...) raised an exception AssertionError: CUDA error in Memcpy2DAsync: initialization error. in smat_cuda/copy.cu:190

[jisraeli]

I'd recommend contacting the DeepBind paper authors about this issue.

The DragoNN package provide a simple toolkit to get started with deep learning modeling of regulatory sequence, you may want to check it out: https://github.com/kundajelab/dragonn

[sthodup1]

Thanks! I'll look into it. I was able to remove the problem be removing the mhltithreading in report.py