coremltools icon indicating copy to clipboard operation
coremltools copied to clipboard

Removing distutils for 3.12 compatibility

Open Cyanosite opened this issue 1 year ago • 1 comments

Distutils has been deprecated in 3.10 and removed in 3.12 (https://docs.python.org/3.10/whatsnew/3.10.html#distutils-deprecated)

For 3.12 compatibility (#2129) I'm proposing to remove distutils.

distutils was only used for its StrictVersion class, this can be replaced by packaging's Version class in every instance, as this library is already a dependency this requires no further packages to be added as dependency.

For the use of the Version class, see: https://packaging.pypa.io/en/latest/version.html

Cyanosite avatar Apr 29 '24 20:04 Cyanosite

Thanks @Cyanosite for this pull request. The change looks good to me.

CI: https://gitlab.com/coremltools1/coremltools/-/pipelines/1288882002

TobyRoseman avatar May 13 '24 16:05 TobyRoseman

  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> [33 lines of output]
      Traceback (most recent call last):
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 112, in get_requires_for_build_wheel
          backend = _build_backend()
                    ^^^^^^^^^^^^^^^^
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 77, in _build_backend
          obj = import_module(mod_path)
                ^^^^^^^^^^^^^^^^^^^^^^^
        File "/Users/cyanosite/Development/coremltools/envs/coremltools-py3.12/lib/python3.12/importlib/__init__.py", line 90, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1310, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1387, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1360, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1331, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
        File "<frozen importlib._bootstrap_external>", line 995, in exec_module
        File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
        File "/private/var/folders/tn/j2ynrhds2zd4fcp7sv3jxwym0000gn/T/pip-build-env-1u1blzlr/overlay/lib/python3.12/site-packages/setuptools/__init__.py", line 16, in <module>
          import setuptools.version
        File "/private/var/folders/tn/j2ynrhds2zd4fcp7sv3jxwym0000gn/T/pip-build-env-1u1blzlr/overlay/lib/python3.12/site-packages/setuptools/version.py", line 1, in <module>
          import pkg_resources
        File "/private/var/folders/tn/j2ynrhds2zd4fcp7sv3jxwym0000gn/T/pip-build-env-1u1blzlr/overlay/lib/python3.12/site-packages/pkg_resources/__init__.py", line 2172, in <module>
          register_finder(pkgutil.ImpImporter, find_on_path)
                          ^^^^^^^^^^^^^^^^^^^
      AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'?
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: subprocess-exited-with-error

× Getting requirements to build wheel did not run successfully.
│ exit code: 1
╰─> See above for output.

note: This error originates from a subprocess, and is likely not a problem with pip.

I'm getting the test pipelines to run locally so you don't have to test my PR every time I submit changes. I ran into the above issue while trying to run the tests. https://stackoverflow.com/questions/77364550/attributeerror-module-pkgutil-has-no-attribute-impimporter-did-you-mean pkgutil.ImpImporter was long deprecated and has been removed in 3.12, due to this I will be trying to do something about this error too. I will get back to you once I have a working solution.

Alternatively I could work on that in a separate PR and just test with 3.10 for now

Cyanosite avatar May 13 '24 21:05 Cyanosite

The test pipeline isn't going to work with Python 3.12. We'll need to update all of our test dependencies to version that support Python 3.12.

I recommend you stick with 3.10 for this pull request.

TobyRoseman avatar May 13 '24 21:05 TobyRoseman

rootdir: /Users/cyanosite/Development/coremltools, configfile: pytest.ini
plugins: flake8-1.0.7, cov-5.0.0, xdist-2.5.0, sugar-1.0.0, mock-3.8.2, timeout-2.3.1, forked-1.6.0
timeout: 600.0s
timeout method: signal
timeout func_only: False
collected 0 items                                                                                                                                    

================================================================== warnings summary ==================================================================
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
<frozen importlib._bootstrap>:914
  <frozen importlib._bootstrap>:914: ImportWarning: _SixMetaPathImporter.find_spec() not found; falling back to find_module()

<frozen importlib._bootstrap>:671
  <frozen importlib._bootstrap>:671: ImportWarning: _SixMetaPathImporter.exec_module() not found; falling back to load_module()

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
------------------------------- generated xml file: /Users/cyanosite/Development/coremltools/build/py-test-report.xml --------------------------------

Results (2.24s):
ERROR: module or package not found: coremltools.test (missing __init__.py?)

I cannot figure out why the pipeline doesn't work locally, seems to be the same error as in the pipeline on gitlab. Flake reports no errors, that runs fine.

Cyanosite avatar May 13 '24 22:05 Cyanosite

Updated CI: https://gitlab.com/coremltools1/coremltools/-/pipelines/1289337174

TobyRoseman avatar May 13 '24 23:05 TobyRoseman

I fixed the remaining importerrors reported in the pipeline, should be good now. Managed to get the tests working locally, the test that failed on the pipeline now runs for me with only one error.

>           assert(_HAS_KMEANS1D)
E           AssertionError

../../envs/coremltools-py3.10/lib/python3.10/site-packages/coremltools/models/neural_network/quantization_utils.py:404: AssertionError

This might be something to do with dependencies on my side rather than the pull request. It might run fine on the pipeline. However the optimizations tests use quite a lot of swap so this might not be something I can run locally often.

Cyanosite avatar May 14 '24 00:05 Cyanosite

Updated CI: https://gitlab.com/coremltools1/coremltools/-/pipelines/1290878130

TobyRoseman avatar May 14 '24 20:05 TobyRoseman