ITK Python v5.3rc03 fails to import ITKImageGrid after upgrade

[tbirdso]

Description

Seeing an issue where ITK v5.3rc03 fails to find the ImageGrid module after upgrading from a previous ITK version such as v5.2.post1. Succeeds on a clean install.

Steps to Reproduce

1. pip install itk==v5.3rc03
2. Launch Python console
3. import itk
4. itk.auto_progress(2) (for module visibility)
5. itk.force_load()

Expected behavior

All modules load without error

Actual behavior

>>> itk.force_load()
Loading ITKPyBase... done
Loading ITKCommon... done
Loading ITKStatistics... done
Loading ITKImageFilterBase... done
Loading ITKTransform... done
Loading ITKMesh... done
Loading ITKImageFunction... done
Loading ITKSpatialObjects... done
Loading ITKSmoothing... done
Loading ITKOptimizers... done
Loading ITKPath... done
Loading ITKImageCompose... done
Loading ITKImageStatistics... done
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/init_helpers.py", line 116, in force_load
    getattr(itk, k)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/lazy.py", line 137, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/base.py", line 98, in itk_load_swig_module
    l_data = itk_base_global_module_data[name]
KeyError: 'ITKImageGrid'
>>> itk.ImageGrid
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/tom/anaconda3/envs/venv-itk/lib/python3.8/site-packages/itk/support/lazy.py", line 130, in __getattribute__
    value = types.ModuleType.__getattribute__(self, attr)
AttributeError: module 'itk' has no attribute 'ImageGrid'

Reproducibility

After upgrading from v5.2.1.post1: 100% After a clean install (uninstall and install all ITK packages with pip): 0%

Versions

ITK 5.3 release candidate 3

Environment

Observed on Windows and Linux in Python 3.8

Additional Information

Observed after direct upgrade with pip install itk==v5.3rc03 on both Windows and Linux. Problem was resolved for both platforms by removing all ITK packages with pip uninstall itk ... and reinstalling with pip install itk.

[tbirdso]

@dzenanz If you have time could you please verify whether you can reproduce? I haven't seen an upgrade path issue for ITK like this before. Simple enough to re-install, but I imagine this may be something we want to resolve before the 5.3 release to avoid forcing existing users to un- and re-install ITK Python modules.

[tbirdso]

@thewtex Ping for visibility

[dzenanz]

I can reproduce it. Windows 10, Python 3.9.6. Going from 5.2.1.post1 to 5.3rc3 gets me:

...
Loading ITKImageStatistics... done
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Program Files\Python39\lib\site-packages\itk\support\init_helpers.py", line 116, in force_load
    getattr(itk, k)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\lazy.py", line 137, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 102, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Program Files\Python39\lib\site-packages\itk\support\base.py", line 98, in itk_load_swig_module
    l_data = itk_base_global_module_data[name]
KeyError: 'ITKImageGrid'

After pip uninstall itk and cleaning up all ITK-related folders in site-packages, then pip install itk==v5.3rc03 gets a properly working ITK.

[tbirdso]

I don't see any commit messages in ITKImageGrid since 5.3rc02 was tagged on Oct 27th that should changed Python module behavior 🤔

ITKImageGrid history

tag history

[dzenanz]

It could be due to some changes in ITKCommon or some Python-related infrastructure.

[tbirdso]

Observed a different error loading ITKFFT this morning that was resolved again by re-installing ITK packages. Haven't tried recreating with an upgrade path.

>>> import itk
>>> itk.__version__
'5.3.0'
>>> itk.auto_progress(2)
>>> itk.Forward1DFFTImageFilter
Loading ITKPyBase... done
Loading ITKCommon... done
Loading ITKStatistics... done
Loading ITKImageFilterBase... done
Loading ITKTransform... done
Loading ITKImageFunction... done
Loading ITKImageGrid... done
Loading ITKFFT... Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\lazy.py", line 137, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 110, in itk_load_swig_module
    l_module = loader.load(swig_module_name)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 259, in load
    l_spec.loader.exec_module(l_module)  # pytype: disable=attribute-error
  File "<frozen importlib._bootstrap_external>", line 783, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\..\ITKFFTPython.py", line 85, in <module>
    from itk.itkForward1DFFTImageFilterPython import *
ModuleNotFoundError: No module named 'itk.itkForward1DFFTImageFilterPython'

[thewtex]

It may be that pip is not resolving package dependencies correctly.

It is best to start from a clean virtual environment / conda environment.

[PranjalSahu]

Faced the same issue with ITK ImageGrid. Works fine in new conda environment.

[tbirdso]

Currently seeing a similar issue again with ITKImageSources in my local environment for v5.3rc04:

>>> import itk
>>> itk.auto_progress(2)
>>> itk.DisplacementFieldTransform
Loading ITKPyBase... done
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\lazy.py", line 138, in __getattribute__
    base.itk_load_swig_module(module, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 96, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 96, in itk_load_swig_module
    itk_load_swig_module(dep, namespace)
  File "C:\Users\tom.birdsong\Anaconda3\envs\venv-itk\lib\site-packages\itk\support\base.py", line 92, in itk_load_swig_module
    l_data = itk_base_global_module_data[name]
KeyError: 'ITKImageSources'

Note that investigation in base.py and site-packages/itk reveal that itk_base_global_module_data is populated via the <module>Config.py files present in itk/Configuration. It appears that ITKImageSourcesConfig.py is not present in my Configuration folder and is not installed along with other config files when I re-install itk-filtering. ITKImageSourcesConfig.py is present in another virtual environment's site-packages/itk/Configuration folder for v5.3rc04, and v5.3rc04 has not failed for me before now, so this does not seem to be as straightforward as the file not being included in the package. Perhaps something being cached strangely?

EDIT: Note that simply copying over this config file does not resolve the issue. No files related to ITKImageSources appear to be present in my site-packages/itk folder.

EDIT2: As before, pip uninstalling all ITK packages and re-installing from cache apparently resolves the problem locally.

[tbirdso]

Note: recently observed by @PranjalSahu via pip install itk in Slicer.

[jcfr]

After comparing the site-packages folder obtained in the following two scenario:

• Install of 5.3rc3 when there is already 5.2.1.post1 installed: site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3
• Install of 5.3rc3 in a clean environment: site-packages-with-itk-5.3rc3

.. we can see that filters added by 5.3rc3 are not installed when upgrading:

[jcfr]

Complete list of missing files after performing the upgrade from 5.2.1.post1 to 5.3rc3:

> itk/Configuration/ITKFFTConfig.py
> itk/Configuration/ITKFFT_snake_case.py
> itk/Configuration/ITKImageGridConfig.py
> itk/Configuration/ITKImageGrid_snake_case.py
> itk/itkBinShrinkImageFilterPython.py
> itk/itkBSplineDownsampleImageFilterPython.py
> itk/itkBSplineUpsampleImageFilterPython.py
> itk/itkChangeInformationImageFilterPython.py
> itk/itkComplexToComplexFFTImageFilterPython.py
> itk/itkConstantPadImageFilterPython.py
> itk/itkCropImageFilterPython.py
> itk/itkCyclicShiftImageFilterPython.py
> itk/itkExpandImageFilterPython.py
> itk/itkFFTPadImageFilterPython.py
> itk/ITKFFTPython.py
> itk/_ITKFFTPython.so
> itk/itkFFTShiftImageFilterPython.py
> itk/itkFFTWComplexToComplexFFTImageFilterPython.py
> itk/itkFFTWForwardFFTImageFilterPython.py
> itk/itkFFTWGlobalConfigurationPython.py
> itk/itkFFTWHalfHermitianToRealInverseFFTImageFilterPython.py
> itk/itkFFTWInverseFFTImageFilterPython.py
> itk/itkFFTWRealToHalfHermitianForwardFFTImageFilterPython.py
> itk/itkFlipImageFilterPython.py
> itk/itkForwardFFTImageFilterPython.py
> itk/itkFullToHalfHermitianImageFilterPython.py
> itk/itkHalfHermitianToRealInverseFFTImageFilterPython.py
> itk/itkHalfToFullHermitianImageFilterPython.py
> itk/ITKImageGridPython.py
> itk/_ITKImageGridPython.so
> itk/itkInterpolateImageFilterPython.py
> itk/itkInverseFFTImageFilterPython.py
> itk/itkMirrorPadImageFilterPython.py
> itk/itkOrientImageFilterPython.py
> itk/itkPadImageFilterBasePython.py
> itk/itkPadImageFilterPython.py
> itk/itkPasteImageFilterPython.py
> itk/itkPermuteAxesImageFilterPython.py
> itk/itkRealToHalfHermitianForwardFFTImageFilterPython.py
> itk/itkRegionOfInterestImageFilterPython.py
> itk/itkResampleImageFilterPython.py
> itk/itkShrinkImageFilterPython.py
> itk/itkSliceBySliceImageFilterPython.py
> itk/itkTileImageFilterPython.py
> itk/itkVnlComplexToComplexFFTImageFilterPython.py
> itk/itkVnlForwardFFTImageFilterPython.py
> itk/itkVnlHalfHermitianToRealInverseFFTImageFilterPython.py
> itk/itkVnlInverseFFTImageFilterPython.py
> itk/itkVnlRealToHalfHermitianForwardFFTImageFilterPython.py
> itk/itkWarpImageFilterPython.py
> itk/itkWrapPadImageFilterPython.py
> itk/itkZeroFluxNeumannPadImageFilterPython.py

List was gathered on Linux in a python3.8 environment

For reference, here is how the list of files copied above was generated:

cd ..
cd site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3
rm -rf itk/__pycache__ itk/support/__pycache__ itk/Configuration/__pycache__
fd . | sort > /tmp/site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3.txt

cd site-packages-with-itk-5.3rc3
fd __pycache__
rm -rf itk/__pycache__ itk/support/__pycache__ itk/Configuration/__pycache__
fd . | sort  > /tmp/site-packages-with-itk-5.3rc3.txt

diff \
  /tmp/site-packages.from-itk-5.2.1.post1-to-itk-5.3rc3.txt \
  /tmp/site-packages-with-itk-5.3rc3.txt | ack "[>|<]"