Multi-GPU support, high-level solve() function, support for non-CULA library alternatives

[roesel]

First of all, thank you for this amazing library! I never expected anything this good to exist already! You are amazing!

Secondly, I would really appreciate an equivalent to the np.linalg.solve() or matlabs mldivide() function. I tried writing my own using the linalg.inv method, but it seems to require CULA even though it does not say so in the docstring.

Is there a way to implement mldivide-like functionality that I have missed, and if not, would that be possible?

Last but not least, is supporting multiple GPUs in store without having CULA? (CULA being not as accessible now...)

Thanks again for the great library! :+1:

[pearcemc]

Not used this particular function myself (and not project affiliated), but the magma bindings are a place to start for a CULA alternative. There are no docs, but magma_sgesv looks promising.

[roesel]

Thank you for your answer. I might try looking into MAGMA, but I have never tried to use it (not do I have MKL, which seems to be necessary). (I also don't think I'll be able to use the function without docs and any previous experience. But thank you for the pointer!)

[lebedov]

A high-level solve function for scikit-cuda would be great, but I haven't gotten around to adding one yet. MAGMA support is also currently incomplete, but I agree that it needs to be brought up to speed to compensate for the reduced availability of CULA (if there are specific wrapper functions you would like me to add to magma.py in the short run, let me know). To get you started with MAGMA, here is a snippet of code demonstrating how to solve a system using the sgesv function @pearcemc alluded to (I just added a wrapper for magma_dgesv_nopiv to 7bd2b42):

import numpy as np
import pycuda.autoinit
import pycuda.gpuarray as gpuarray

import skcuda.magma as magma

np.random.seed(0)
n = 1000

# Note that the matrices have to either be stored in column-major form,                      
# otherwise MAGMA will compute the solution for the system dot(a.T, x) == b:                 
a = np.asarray(np.random.rand(n, n), order='F')
b = np.asarray(np.random.rand(n), order='F')

# Compute solution on the CPU:                                                               
x = np.linalg.solve(a, b)

# Compute the solution on the GPU - b_gpu is subsequently overwritten with the solution:     
a_gpu = gpuarray.to_gpu(a)
b_gpu = gpuarray.to_gpu(b)
magma.magma_init()
magma.magma_dgesv_nopiv_gpu(n, 1, int(a_gpu.gpudata), n, int(b_gpu.gpudata), n)
magma.magma_finalize()

# Check that the solutions match:                                                            
print np.allclose(x, b_gpu.get())

You will, of course, have to build/install MAGMA's shared library and make sure that the library loader knows where it is in order to use the wrapper.

MAGMA does contain some support for multiple GPUs, but I don't believe it provides high-level solvers that automatically use multiple GPUs yet (as of version 1.6.2).

You might also want to check out the CUSOLVER library included in CUDA 7.0 - scikit-cuda has wrappers for various functions in it.

[roesel]

Thank you sir!

I will invest some time into this and see how far I can get. Thank you for all the pointers and the example and I hope I will be able to achieve some results even with my limited knowledge.

I'll leave this open for now for perhaps other tips and questions regarding this problem. Thanks again!

[roesel]

OK, I feel a little stupid as I tried to install the current version of scikit-cuda from the repo, but now when I run my code, it doesn't even get past the imports

File "/home/david/ice2/cudastuff.py", line 4, in <module>
    from skcuda import linalg, misc
File "/home/david/anaconda/lib/python2.7/site-packages/scikit_cuda-0.5.1-py2.7.egg/skcuda/__init__.py", line 9, in <module>
    from .info import __doc__
SystemError: Parent module 'skcuda' not loaded, cannot perform relative import

even though I think the python setup.py install was successful. Maybe I'm just doing a newbie mistake somewhere, but I thought I went precisely according to the documentation.

[lebedov]

Did you have an existing installation of scikit-cuda somewhere? I can't replicate that error if I install the latest revision from scratch in a conda environment.

[roesel]

I did. I ran pip uninstall scikit-cuda and then tried to install the one form the repo using the python setup.py install method.

[lebedov]

What version of Python are you using?

[roesel]

Python 2.7.10 |Anaconda 2.3.0 (64-bit)| (default, May 28 2015, 17:02:03)
[GCC 4.4.7 20120313 (Red Hat 4.4.7-1)] on linux2

[lebedov]

What is your PYTHONPATH env variable set to? If you run python -c "import sys,pprint; pprint.pprint(sys.path)", what do you get? Also, do you have the latest version of setuptools installed?

[roesel]

If I run what you posted, I get (this is with the stable scikit-cuda installed by pip install scikit-cuda):

['',
 '/home/david/anaconda/lib/python2.7/site-packages/pycuda-2015.1.2-py2.7-linux-x86_64.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/appdirs-1.4.0-py2.7.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/pytools-2015.1.2-py2.7.egg',
 '/home/david/anaconda/lib/python27.zip',
 '/home/david/anaconda/lib/python2.7',
 '/home/david/anaconda/lib/python2.7/plat-linux2',
 '/home/david/anaconda/lib/python2.7/lib-tk',
 '/home/david/anaconda/lib/python2.7/lib-old',
 '/home/david/anaconda/lib/python2.7/lib-dynload',
 '/home/david/anaconda/lib/python2.7/site-packages',
 '/home/david/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/cryptography-0.9.1-py2.7-linux-x86_64.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/setuptools-18.0.1-py2.7.egg']

After installing from the repo:

['',
 '/home/david/anaconda/lib/python2.7/site-packages/pycuda-2015.1.2-py2.7-linux-x86_64.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/appdirs-1.4.0-py2.7.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/pytools-2015.1.2-py2.7.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/scikit_cuda-0.5.1-py2.7.egg',
 '/home/david/anaconda/lib/python27.zip',
 '/home/david/anaconda/lib/python2.7',
 '/home/david/anaconda/lib/python2.7/plat-linux2',
 '/home/david/anaconda/lib/python2.7/lib-tk',
 '/home/david/anaconda/lib/python2.7/lib-old',
 '/home/david/anaconda/lib/python2.7/lib-dynload',
 '/home/david/anaconda/lib/python2.7/site-packages',
 '/home/david/anaconda/lib/python2.7/site-packages/Sphinx-1.3.1-py2.7.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/cryptography-0.9.1-py2.7-linux-x86_64.egg',
 '/home/david/anaconda/lib/python2.7/site-packages/setuptools-18.0.1-py2.7.egg']

And the second question (though obvious from last line of previous code):

$ easy_install --version
setuptools 18.0.1

Only invest time in this if the bug itself interests you. It's not a big deal for me. :) Thank you for your time!

[lebedov]

Strange - I'm using the same version of Python (via miniconda) and setuptools, but I can't reproduce the error. You might also want to try installing scikit-cuda and its dependencies in a new conda environment (judging by the error message, you appear to have installed it in the root conda environment of your Anaconda installation).

[roesel]

Man I'm really confused. I might give that another try, maybe a fresh installation of anaconda will do the trick, who knows. Thanks for your time!

[akusok]

Worked for me with Anaconda and MAGMA 1.7, did not work with MAGMA 2.0

I had a problem with missing symbol in "magma.py" but it was only one weird function. Deleted it from the file "magma.py" in the installation folder; the rest worked.

P.S. It would be great to have "ds..." mixed-precision solvers, they are a key point in MAGMA and super useful on consumer-grade GPUs with poor 64-bit performance.

P.P.S. Thanks again for the great library!