tlambert03
On Windows, gputools and libcudaDeconv compete for GPU: OSError: [WinError -529697949]
tracking a bug seen on some windows computers (definitely seen on win10, maybe win7) in which importing gputools causes a crash when trying to access shared functions in libcudaDeconv.
>>> import llspy
>>> import gputools # comment out this line and it succeeds
>>> previewPath = 'basic_sample'
>>> otfdir = 'OTFs'
>>> E = llspy.LLSdir(previewPath, True)
>>> llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
"""
WARNING:gputools.config.myconfigparser | [Errno 2] No such file or directory: '~/.gputools'
output nz=84
cufftPlan3d() c2r failed
OSError Traceback (most recent call last)
<ipython-input-1-c354b9e601b9> in <module>()
5
6 E = llspy.LLSdir(previewPath, True)
----> 7 llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
~\Dropbox (HMS)\Python\llspy\llspy\llsdir.py in preview(exp, tR, cR, **kwargs)
197 for i, d in enumerate(zip(stacks, P.otfs)):
198 stk, otf = d
--> 199 stacks[i] = quickDecon(stk, otf, **opts)
200 else:
201 # deconvolution does deskewing and cropping, so we do it here if we're
~\Dropbox (HMS)\Python\llspy\llspy\libcudawrapper.py in quickDecon(im, otfpath, savedeskew, **kwargs)
229 return decon_result, deskew_result
230 else:
--> 231 decon_result = RL_decon(im, savedeskew=False, **kwargs)
232 RL_cleanup()
233 return decon_result
~\Dropbox (HMS)\Python\llspy\llspy\libcudawrapper.py in RL_decon(im, background, nIters, shift, savedeskew, rescale, **kwargs)
257 im = im.astype(np.uint16)
258 RL_interface(im, nx, ny, nz, decon_result, deskew_result,
--> 259 background, rescale, savedeskew, nIters, shift)
260
261 if savedeskew:
OSError: [WinError -529697949] Windows Error 0xe06d7363
On the other hand, if gputools is imported after the first call to libcudaDeconv, then gputools fails to get resources
>>> import llspy
>>> import gputools # comment out this line and it succeeds
>>> previewPath = 'basic_sample'
>>> otfdir = 'OTFs'
>>> E = llspy.LLSdir(previewPath, True)
>>> llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
output nz=84
background=0.000000
Iteration 0
Iteration 1
Iteration 2
Iteration 3
Iteration 4
Iteration 5
Iteration 6
Iteration 7
Iteration 8
Iteration 9
>>> import gputools
WARNING:gputools.config.myconfigparser | [Errno 2] No such file or directory: '~/.gputools'
>>> gputools.init_device()
"""
...choosing the best from the rest
---------------------------------------------------------------------------
RuntimeError Traceback (most recent call last)
<ipython-input-1-45da538f8b12> in <module>()
7 llspy.llsdir.preview(E, 0, 0, otfDir=otfdir)
8 import gputools
----> 9 gputools.init_device()
10
~\Anaconda3\envs\llsbuild\lib\site-packages\gputools\config\config.py in init_device(**kwargs)
54 ....
---> 56 new_device = OCLDevice(**kwargs)
57
58 # just change globals if new_device is different from old
~\Anaconda3\envs\llsbuild\lib\site-packages\gputools\core\ocldevice.py in __init__(self, id_platform, id_device, use_gpu, print_info, context_properties, initCL, **kwargs)
38 print_info=print_info,
39 context_properties=context_properties,
---> 40 **kwargs)
41
42 @classmethod
~\Anaconda3\envs\llsbuild\lib\site-packages\gputools\core\ocldevice.py in init_cl(self, id_platform, id_device, use_gpu, print_info, context_properties)
92
93 self.context = pyopencl.Context(devices=[device],
---> 94 properties=context_properties)
95
96 self.device = device
~\Anaconda3\envs\llsbuild\lib\site-packages\pyopencl\cffi_cl.py in __init__(self, devices, properties, dev_type, cache_dir)
790 # TODO parameter order? (for clobj_list)
791 _handle_error(_lib.create_context(_ctx, c_props,
--> 792 num_devices, _devices))
793
794 else:
~\Anaconda3\envs\llsbuild\lib\site-packages\pyopencl\cffi_cl.py in _handle_error(error)
643 _lib.free_pointer(error.msg)
644 _lib.free_pointer(error)
--> 645 raise e
646
647 # }}}
RuntimeError: clCreateContext failed: OUT_OF_RESOURCES
running CUDA DeviceQuery shows that the GPU does not seem to be running in exclusive mode:
Compute Mode:
< Default (multiple host threads can use ::cudaSetDevice() with device simultaneously) >
but nonetheless, the two don't work together well... Not sure where to find the equivalent setting in opencl/clinfo
doesn't seem to be a problem on mac or linux...
not a fix, but a better error message introduced in b19c9bb15d589464df666cbc8537f91ee35c2456 and 0.2.3, with the option to disable spimagine (which fixes the problem, but ... obviously... disables spimagine)
Would it be possible to pin the gpuTools to one gpu, and use another for decon? This would be nice for our system since we have regularly turn uncheck our Quadro for decon since it has a smaller memory size, and just use the tesla
i'll try to look in to that. If the main issue is just that it's annoying to have to uncheck the quadro every time you start the program, it would relatively easy for me to just add that to settings that get saved automatically (so that it remembered your setting from session to session)... but i fear that the problem with gputools playing poorly with the libcudaDeconv library goes deeper than that (in the past, if I even imported them together, stuff broke on some Win10 systems), so you might still not be able to use the spimagine viewer if you're having this annoying WinError problem
that said, I don't think I ever really looked into an approach where gputools and libcudadeconv got their own GPU, so I'll try to see how hard it will be (first I need to remember some things about how it works!)
Thanks. I was just thinking of a way to get the preview to work with spimagine, it's not an issue of keeping the box checked.
Using the new cudaDeconv that Lin had provided back in june, I made some modifications in a forked repo to utilize that version correctly for the bleach correction and rotation, since some of the flags had changed or were added.