BUG: `v5.3rc04.post2` breaks backwards compatibility with external module Python packages

[tbirdso]

Description

Python module loading change introduced in https://github.com/InsightSoftwareConsortium/ITK/commit/835dc01388d22c4b4c9a46b01dbdfe394ec23511 appears to break backwards compatibility with ITK Python external modules targeting an older version of ITK.

Steps to Reproduce

1. Install external module package. Tested with itk-meshtopolydata, itk-elastix, itk-ultrasound
2. In a Python console try to force load modules with verbosity on

> import itk
> itk.auto_progress(2)
> itk.force_load()

Expected behavior

Modules load without error. With itk-meshtopolydata==0.8.4 which is built against the ITK Python v5.3rc04.post2 tag including the relevant commit:

>>> itk.force_load()
Loading ITKPyBase... done
Loading ITKCommon... done
...
Loading ITKIOTransformMatlab... done
done
Loading MeshToPolyData... done
Loading ITKDeconvolution... done
Loading ITKWatersheds... done
...
Loading ITKIOSpatialObjects... done
Loading ITKVTK... done

Actual behavior

Error on load. With itk-meshtopolydata==0.8.3 which is built against the v5.3rc04.post1 ITK Python tag not including the relevant commit:

>>> itk.force_load()
...
Loading ITKIOTransformMatlab... done
done
Loading MeshToPolyData... Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\init_helpers.py", line 116, in force_load
    getattr(itk, k)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\lazy.py", line 138, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 132, in itk_load_swig_module
    l_module = loader.load(swig_module_name)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 291, in load
    l_spec.loader.exec_module(l_module)  # pytype: disable=attribute-error
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\..\MeshToPolyDataPython.py", line 13, in <module>
    from . import _MeshToPolyDataPython
AttributeError: PyCapsule_Import "_ITKCommonPython._C_API" is not valid

Reproducibility

100%

Versions

ITK version: itk==v5.3rc04.post2

Environment

Windows 10 Python 3.8.5

Incompatible: itk-meshtopolydata==v0.8.3 Incompatible: itk-ultrasound==v0.5.6 Incompatible: itk-elastix==0.14.1

Additional Information

My understanding is that https://github.com/InsightSoftwareConsortium/ITK/commit/835dc01388d22c4b4c9a46b01dbdfe394ec23511 is an important fix. This behavior may be determined to be an acceptable tradeoff with a heightened focus on getting remote modules updated for v5.3rc04.post2 or later.

cc @thewtex @jcfr

[tbirdso]

@jcfr @thewtex Could one of you please comment on the severity of this bug in regards to the v5.3 roadmap? I have not been able to allocate time to investigating whether a fix is possible without reverting https://github.com/InsightSoftwareConsortium/ITK/commit/835dc01388d22c4b4c9a46b01dbdfe394ec23511.

[jcfr]

To maintain compatibility (either backward or explicitly with dask), I am wondering if is possible to introduce a function allowing to change name of the ITK capsule using PyCapsule_SetName ^1.

[jcfr]

either backward or explicitly with dask

Understand how prevalent each use case are:

• integration with dask

vs

• use of newer module within older ITK

may help.

[thewtex]

Remote modules will need to be updated to build against the latest ITK -- we do not have binary compatibility with remote modules across ITK versions.

[thewtex]

Remote modules should update to ITK 5.3.0. See https://github.com/InsightSoftwareConsortium/ITKRemoteModuleBuildTestPackageAction for help with this