CULA vs (CUDA Toolkit)cuSOLVER vs MAGMA

[kevincopps]

Thanks for working on a great library.

I'm interested in using double precision GPU accelerated QR and SVD capabilities in Python---primarily on Linux and Mac platforms. I just noticed that it seems the CULA libraries implementing this functionality in 64 bit floats require a commercial license. In addition, the latest two versions of CULA (R17, R18) don't list a Mac OSX download at all... kinda confusing.

So, question: Is most or all of this solver functionality in CULA now made obsolete by new versions of the CUDA Toolkit which include the cuSOLVER API? As of 2015-07-15, a glance at some of the functions in the CUDA Toolkit 7.0 cuSOLVER looked almost identical to those in CULA. What's your take on the relationship there? Any plans to use cuSOLVER from CUDA Toolkit instead of CULA in scikit-cuda?

As I was about to write this, I just read about the MAGMA library and saw you had some initial code for MAGMA interfaces already in the repo. Will you just shift to MAGMA instead for all the linear algebra solvers?

[lebedov]

I would like to have the high-level functions (i.e., in the linalg module) be able to use CUSOLVER or MAGMA (if available) given that obtaining access to CULA is no longer as easy as it used to be, although MAGMA support is relatively low priority for me now owing to other ongoing projects I am working on.

I just added wrappers for the geqrf functions (QR factorization) in CUSOLVER to the master branch; as you probably noticed, there currently isn't any high-level interface to QR factorization in scikit-cuda yet (although you can try constructing one using the Cholesky factorization high-level interface as a starting point). I'll try to tweak the high-level linalg.svd to optionally use CUSOLVER's SVD functions.

[kevincopps]

Great! I'll take a look and try them out.

On Friday, July 17, 2015, Lev Givon [email protected] wrote:

I would like to have the high-level functions (i.e., in the linalg module) be able to use CUSOLVER or MAGMA (if available) given that obtaining access to CULA is no longer as easy as it used to be, although MAGMA support is relatively low priority for me now owing to other ongoing projects I am working on.

I just added wrappers for the geqrf functions (QR factorization) in CUSOLVER to the master branch; as you probably noticed, there currently isn't any high-level interface to QR factorization in scikit-cuda yet (although you can try constructing one using the Cholesky factorization high-level interface as a starting point). I'll try to tweak the high-level linalg.svd to optionally use CUSOLVER's SVD functions.

— Reply to this email directly or view it on GitHub https://github.com/lebedov/scikit-cuda/issues/127#issuecomment-122278876 .[image: Web Bug from https://github.com/notifications/beacon/ACfJ01Z6l-y7rfbM8mKeKguLnp5efFWNks5oeP8qgaJpZM4FZbgq.gif]

[untom]

Just as a head's up: I ran into a bad memory-leak within cuSOLVER's cholesky factorization, so depending on your needs, you might want to avoid those calls: they leak GPU memory on each call. nvidia confirmed the bug and said it would be fixed in the next CUDA release.

[kevincopps]

Thanks! You're on top of things. I appreciate you letting us know.

On Wednesday, July 22, 2015, untom [email protected] wrote:

Just as a head's up: I ran into a bad memory-leak within cuSOLVER's cholesky factorization, so depending on your needs, you might want to avoid those calls: they leak GPU memory on each call. nvidia confirmed the bug and said it would be fixed in the next CUDA release.

— Reply to this email directly or view it on GitHub https://github.com/lebedov/scikit-cuda/issues/127#issuecomment-123728096 .[image: Web Bug from https://github.com/notifications/beacon/ACfJ06vrAkWAumPdbVd1IsOJfewvwBhQks5of5a8gaJpZM4FZbgq.gif]

[erichson]

I guess cuSOLVER won't replace CULA very soon. So far the methods are seeming to be very limited. Just to mention SVD, here only the 'full' SVD decomposition is available, but in practice the 'reduced/economic' decomposition might be very important too.

But let's see what the future brings. :)

[lebedov]

I just added support for computing the full SVD with CUSOLVER rather than CULA if so desired.

[paoloviviani]

I just stumbled in this issue and I hope I can contribute, even after some months. As far as I know CuSolver only implements GESVD, while Magma implements GESDD, which is much faster. I tested Cusolver against Magma myself and they are not even close. Moreover Magma's interface is more lapack-like than cusolver's and it is easier to use it as a drop-in replacement of LAPACK. I don't know however where it stands when compared to CULA.

[lebedov]

@paoloviviani, there are wrappers for MAGMA's GESVD and GESDD functions in skcuda/magma.py (although one obviously must install the MAGMA shared library manually to use them). If you would like to contribute, it would be great to try extending skcuda.linalg.svd to permit the use of MAGMA; let me know if you encounter issues with the wrappers.

[simbaforrest]

@untom Anyone know whether cuSOLVER's Cholesky factorization still suffers for the memory leak in CUDA 7.5?

[untom]

No, IIRC it was fixed in CUDA 7.5