ITK
ITK copied to clipboard
InsightSoftwareConsortium
DLL Load failed while import _ITKCommonPython error
Description
Getting error while reading mesh using ITK.
---------------------------------------------------------------------------
ImportError Traceback (most recent call last)
Input In [3], in <cell line: 1>()
----> 1 a = itk.meshread("C:\\Users\\pranj\\Downloads\\testpoly.vtk", itk.D)
2 print(a)
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\extras.py:1080, in meshread(filename, pixel_type, fallback_only)
1078 except (KeyError, itk.TemplateTypeError):
1079 pass
-> 1080 TemplateReaderType = itk.MeshFileReader
1081 io_filename = f"{filename}"
1082 increase_dimension = False
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\lazy.py:137, in LazyITKModule.__getattribute__(self, attr)
135 module = self.__belong_lazy_attributes[attr]
136 namespace = {}
--> 137 base.itk_load_swig_module(module, namespace)
138 self.loaded_lazy_modules.add(module)
139 for k, v in namespace.items():
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:102, in itk_load_swig_module(name, namespace)
100 deps = l_data.get_module_dependencies()
101 for dep in deps:
--> 102 itk_load_swig_module(dep, namespace)
104 if itkConfig.ImportCallback:
105 itkConfig.ImportCallback(name, 0)
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:102, in itk_load_swig_module(name, namespace)
100 deps = l_data.get_module_dependencies()
101 for dep in deps:
--> 102 itk_load_swig_module(dep, namespace)
104 if itkConfig.ImportCallback:
105 itkConfig.ImportCallback(name, 0)
[... skipping similar frames: itk_load_swig_module at line 102 (2 times)]
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:102, in itk_load_swig_module(name, namespace)
100 deps = l_data.get_module_dependencies()
101 for dep in deps:
--> 102 itk_load_swig_module(dep, namespace)
104 if itkConfig.ImportCallback:
105 itkConfig.ImportCallback(name, 0)
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:110, in itk_load_swig_module(name, namespace)
107 # SWIG-generated modules have 'Python' appended. Only load the SWIG module
108 # if we haven't already.
109 loader = LibraryLoader()
--> 110 l_module = loader.load(swig_module_name)
112 # OK, now the modules on which this one depends are loaded and
113 # template_feature-instantiated, and the SWIG module for this one is also loaded.
114 # We're going to put the things we load and create in two places: the
(...)
120 # stomp on an existing 'swig' namespace, nor do we want to share 'swig'
121 # namespaces between this_module and namespace.
123 if namespace is None:
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\base.py:259, in LibraryLoader.load(self, name)
257 # since version 3.4: Use importlib.util.find_spec() instead.
258 l_spec = importlib.util.find_spec(name)
--> 259 l_spec.loader.exec_module(l_module) # pytype: disable=attribute-error
260 return l_module
261 finally:
File <frozen importlib._bootstrap_external>:843, in exec_module(self, module)
File <frozen importlib._bootstrap>:219, in _call_with_frames_removed(f, *args, **kwds)
File ~\anaconda3\envs\mypython3\lib\site-packages\itk\support\..\ITKCommonPython.py:13, in <module>
11 # Import the low-level C/C++ module
12 if __package__ or "." in __name__:
---> 13 from . import _ITKCommonPython
14 else:
15 import _ITKCommonPython
ImportError: DLL load failed while importing _ITKCommonPython: The specified module could not be found.
This could potentially create problems when working with Coiled clusters.
Steps to Reproduce
Expected behavior
Actual behavior
Reproducibility
100%
Versions
5.3rc4, 5.3rc3, 5.3rc2.
Environment
In a Windows 10 Home VM where the host machine is Ubuntu 20. When ITK is installed in a conda environment.
Additional Information
5.3rc1 and 5.2.1post1 work fine.
@PranjalSahu Could you please try running the small script below and report back on whether it succeeds? This will help with understanding whether the ITKCommon module is present and can be loaded at all.
import itk
itk.Image # implicitly load ITKCommon module
Coiled error message
ImportError Traceback (most recent call last)
<ipython-input-46-b727c8b9b1bb> in <module>()
----> 1 get_ipython().run_cell_magic('time', '', '\nl = compute(numParam)')
16 frames
/usr/local/lib/python3.7/dist-packages/IPython/core/interactiveshell.py in run_cell_magic(self, magic_name, line, cell)
2115 magic_arg_s = self.var_expand(line, stack_depth)
2116 with self.builtin_trap:
-> 2117 result = fn(magic_arg_s, cell)
2118 return result
2119
<decorator-gen-53> in time(self, line, cell, local_ns)
/usr/local/lib/python3.7/dist-packages/IPython/core/magic.py in <lambda>(f, *a, **k)
186 # but it's overkill for just that one bit of state.
187 def magic_deco(arg):
--> 188 call = lambda f, *a, **k: f(*a, **k)
189
190 if callable(arg):
/usr/local/lib/python3.7/dist-packages/IPython/core/magics/execution.py in time(self, line, cell, local_ns)
1191 else:
1192 st = clock2()
-> 1193 exec(code, glob, local_ns)
1194 end = clock2()
1195 out = None
<timed exec> in <module>()
/usr/local/lib/python3.7/dist-packages/dask/base.py in compute(traverse, optimize_graph, scheduler, get, *args, **kwargs)
571 postcomputes.append(x.__dask_postcompute__())
572
--> 573 results = schedule(dsk, keys, **kwargs)
574 return repack([f(r, *a) for r, (f, a) in zip(results, postcomputes)])
575
/usr/local/lib/python3.7/dist-packages/distributed/client.py in get(self, dsk, keys, workers, allow_other_workers, resources, sync, asynchronous, direct, retries, priority, fifo_timeout, actors, **kwargs)
2992 should_rejoin = False
2993 try:
-> 2994 results = self.gather(packed, asynchronous=asynchronous, direct=direct)
2995 finally:
2996 for f in futures.values():
/usr/local/lib/python3.7/dist-packages/distributed/client.py in gather(self, futures, errors, direct, asynchronous)
2150 direct=direct,
2151 local_worker=local_worker,
-> 2152 asynchronous=asynchronous,
2153 )
2154
/usr/local/lib/python3.7/dist-packages/distributed/utils.py in sync(self, func, asynchronous, callback_timeout, *args, **kwargs)
308 else:
309 return sync(
--> 310 self.loop, func, *args, callback_timeout=callback_timeout, **kwargs
311 )
312
/usr/local/lib/python3.7/dist-packages/distributed/utils.py in sync(loop, func, callback_timeout, *args, **kwargs)
374 if error:
375 typ, exc, tb = error
--> 376 raise exc.with_traceback(tb)
377 else:
378 return result
/usr/local/lib/python3.7/dist-packages/distributed/utils.py in f()
347 future = asyncio.wait_for(future, callback_timeout)
348 future = asyncio.ensure_future(future)
--> 349 result = yield future
350 except Exception:
351 error = sys.exc_info()
/usr/local/lib/python3.7/dist-packages/tornado/gen.py in run(self)
1131
1132 try:
-> 1133 value = future.result()
1134 except Exception:
1135 self.had_exception = True
/usr/local/lib/python3.7/dist-packages/distributed/client.py in _gather(self, futures, errors, direct, local_worker)
2007 exc = CancelledError(key)
2008 else:
-> 2009 raise exception.with_traceback(traceback)
2010 raise exc
2011 if errors == "skip":
/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/utils.py in offload()
/opt/conda/envs/coiled/lib/python3.7/concurrent/futures/thread.py in run()
/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/utils.py in <lambda>()
/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/worker.py in _deserialize()
/opt/conda/envs/coiled/lib/python3.7/site-packages/distributed/protocol/pickle.py in loads()
/opt/conda/envs/coiled/lib/python3.7/site-packages/itk/itkCompositeTransformPython.py in <module>()
ImportError: PyCapsule_Import could not import module "_ITKCommonPython"
A few conditions that may be problematic here:
/opt/conda/envs/coiled/lib/python3.7/site-packages
is being used with
/usr/local/lib/python3.7/dist-packages/
-- can the same environment be used?
----> 1 get_ipython().run_cell_magic('time'
Can we avoid running in ipython, at least for the test, so there are not any fancy dask/ipython issues?
I have got it to work on the coiled cluster by passing the dict as a function argument instead of the mesh object. So we can build the pipeline with this.
@delayed
def perform_sum1(mesh_dict):
import itk
mesh = itk.mesh_from_dict(mesh_dict)
return mesh['dimension']
I will also check how to get it to work when directly passing objects.
@PranjalSahu Does this mean you were able to import ITK modules successfully? It looks like mesh_from_dict is loading ITKMesh:
def mesh_from_dict(mesh_dict: Dict) -> "itkt.Mesh":
"""Deserialize an dictionary representing an itk.Mesh object."""
import itk
MeshType = mesh_type_from_wasm_type(mesh_dict["meshType"]) # I assume this returns a type like itk.Mesh[itk.F,3]
mesh = MeshType.New() # Mesh instantiation requires ITKMesh DLL to be loaded
...
You could verify that modules are loading by finding the ITK extras.py being used by your environment and inserting itk.auto_progress(2) to get printouts from module loading.
@PranjalSahu Another script to try:
>> import itk
>> itk.auto_progress(2)
>> itk.Image[itk.D,3].New()
EDIT: above script fails on Pranjal's VM; ITKPyBase imports successfully and then ITKCommon is not found
Adiditional information
Edition Windows 10 Home
Version 21H2
Installed on 3/19/2022
OS build 19044.1645
Experience Windows Feature Experience Pack 120.2212.4170.0
For reference Pranjal is using a Python 3.8.12 virtual environment on his Win10 Home virtual machine.
EDIT: Note that there are two machines under discussion here, an individual Windows 10 VM and generalized Dask clusters (running a Debian Linux image) managed with the Coiled Python package. Both are showing the same error string where the ITKCommon library cannot be successfully loaded.
Coiled / Dask Cluster: AWS Debian Linux instance spawned via Coiled, Python 3.7
Local: Windows 10 Home, Python 3.8.12
During our discussion @PranjalSahu found an apparent workaround where, on one runner, first executing an operation with a mesh dictionary and then with an itk.Mesh object is successful and does not show a loading error. It looks like there is not an easy way to SSH directly into the runner for validating packages and testing functionality with printouts. @PranjalSahu to investigate reproducing the error on a separate EC2 instance launched from the same Docker image. This issue takes on lessened priority as work is no longer blocked.
Additional Information:
The method works when we first perform computation using itk.dict_from_mesh. Next time even directly passing mesh object as function argument works.
@delayed
def perform_sum5(img_temp):
import itk
import numpy as np
# here img_temp is a dict object which we convert to mesh inside the method
mesh = itk.mesh_from_dict(img_temp)
return np.sum(mesh['points'])
@delayed
def perform_sum6(img_temp):
import itk
import numpy as np
# here img_temp is a mesh object
return np.sum(img_temp['points'])
Here calling perform_sum6 in a fresh was worker fails. But first calling perform_sum5 and then perform_sum6 works.
It looks like there is not an easy way to SSH directly into the runner for validating packages and testing functionality with printouts.
@jrbourbeau @mrocklin any tips to SSH into a Coiled Dask work for debugging?
xref https://github.com/InsightSoftwareConsortium/ITKPythonPackage/issues/194
@PranjalSahu Could you please try running the small script below and report back on whether it succeeds? This will help with understanding whether the ITKCommon module is present and can be loaded at all.
import itk itk.Image # implicitly load ITKCommon module
If it's helpful to know, I can run this without error in my jupyter notebook (and I think I might be running into the same problem, in a different context, discussion here https://github.com/dask/dask-blog/issues/138)
ImportError: PyCapsule_Import could not import module "_ITKCommonPython"
This is addressed in: https://github.com/InsightSoftwareConsortium/ITK/pull/3494
However, we likely still need to import itk in the Dask function due to issues with pickling of builtin-functions :-(.
ImportError: DLL load failed while importing _ITKCommonPython: The specified module could not be found.
I think this (Anaconda + Windows) is likely a different issue. It may be that there are different Windows runtime DLLs that cannot be found.
I tried mamba / conda-forge environments:
- 3.7: Worked
- 3.8: Failed
- 3.9: Worked
- 3.10: Worked
A hint from @jcfr, there was a recent fix in CMake: https://github.com/Kitware/CMake/commit/787ab7ff20ca74b57cfd3c18b0e92be1844ba244. I will try using a newer CMake for the builds.
Status update: itk-5.3rc4.post2 incorporates #3494, and dask support is stable! :tada: (caveat: add import itk in delayed dask functions).
The Anaconda Windows issue is separate, we could address with conda-forge native package, perhaps via clang module optimized packages (I am investigating this).
The issue still seems to persist under Anaconda Windows, is someone intending to fix this?
Resolved in itk-5.3.0 per https://github.com/InsightSoftwareConsortium/ITKPythonPackage/issues/194