nexB
Python 3.14 support and dependency pinning and deprecation warning policies
Description
Please leave a brief description of the bug or feature request:
Fedora is proud to be an early adopter of Python 3.14 as part of our goal to help the ecosystem by testing and integrating new Python versions, and I worked on getting all dependencies functional and addressing test failures in the scancode stack. I am creating this issue to track supporting Python 3.14 officially upstream. We only run a subset of upstream scancode tests, but this reflects new failures after the Python 3.14 update.
There are also some concerns I have with dependency pinning and deprecation warning practices in the aboutcode projects that I mentioned that created extra friction when updating to Python 3.14. See the Commentary below.
scancode issues
-
[x] https://github.com/aboutcode-org/scancode-toolkit/pull/4539 that I filed earlier today was needed due to ast module removals.
-
[ ] We ignored the remaining test failures in our scancode build:
# ERROR: 'sbin/fstrim' is a link to an absolute path # These two are due to the tarfile module defaulting to the data filter in py3.14. # Let upstream figure out what to do about this issue. test_can_get_installed_system_packages_with_license_from_alpine_container_layer test_can_scan_installed_system_package_in_alpine_container_layer # These test failures have to do with the handling of tempfiles in the CLI # and don't seem particularly important. Disable them for now test_scan_keep_temp_files_is_false_by_default test_scan_keep_temp_files_keeps_files and not -
[ ] https://github.com/aboutcode-org/scancode-toolkit/issues/4540 is not directly caused by Python 3.14 but also impacted us while trying to update scancode to the latest version.
Commentary: Both the ast and tarfile issues were due to known deprecation that upstream seemingly ignored, so perhaps aboutcode should consider adopting a more proactive approach to addressing deprecation warnings (such as running pytest with -W error) and avoid pinning old dependency versions [^1]. I'm not sure what's going on with the temp_files tests. I didn't take the time to debug them, and I'm not sure if the issue is with the Fedora build environment.
dependency issues
- [x] I replaced
pkginfo2withpkginfoin scancode with a downstream patch.pkginfo2seems to be an under-maintained fork ofpkginfoand has failures with new setuptools versions that prevented it from being rebuilt for Python 3.14. These were reported in https://github.com/aboutcode-org/pkginfo2/issues/5 but never addressed upstream. - [x] https://github.com/aboutcode-org/commoncode/issues/88 requires disabling an additional test in our commoncode package. This was also caused by tarfile changes.
- [x] Issues with beartype prevented spdx-tools from being rebuilt. These were fixed in https://github.com/beartype/beartype/issues/537, but a new upstream release has not yet been created.
- [ ] Once we updated beartype to the latest git snapshot, I backported https://github.com/spdx/tools-python/commit/6817ca7056cadfd71c5619b7d75452461e4bbb18 for beartype compatibility and submitted https://github.com/spdx/tools-python/pull/858 to spdx-tools to fix other issues that came up in the spdx-tools package. The spdx-tools maintainers have not yet applied my patch.
Commentary: The beartype issues were understandable due to major typing changes in Python 3.14. I'm not sure about the state of the pkginfo2 project or spdx-tools.
How To Reproduce
Tell us how to reproduce the issue.
Run scancode tests in an environment with Python 3.14. Make sure to install the latest git snapshots of https://github.com/gotmax23/tools-python/tree/py314-annotations and https://github.com/beartype/beartype that have the patches mentioned.
System configuration
For bug reports, it really helps us to know:
- What OS are you running on? (Windows/MacOS/Linux) Fedora Linux Rawhide (44)
- What version of scancode-toolkit was used to generate the scan file? 32.4.1
- What installation method was used to install/run scancode? (pip/source download/other) source
Thank you for reading my wall of text and for all the work you do on scancode!
[^1]: EDIT: To be clear, I was referring to pinning of test dependencies for CI in requirements.txt, not the dependencies in the packaging metadata (setup.cfg) which are specified properly without excessive pinning for the most part. After looking again, it seems like requirements.txt does get updated on a somewhat regular basis but importlib-metadata was just held back in https://github.com/aboutcode-org/scancode-toolkit/commit/97bc9b5a6e752d06124280c66d5965b6be723293. Using something like dependabot or a Github Actions scheduled pipeline to automatically submit PRs to bump requirements.txt is still a good idea.
Great to see Python 3.14 support!
I can't even install it on Python 3.13 😅
uvx scancode-toolkit
Built ftfy==4.4.3 × Failed to build `bitarray==0.9.3`
├─▶ The build backend returned an error
╰─▶ Call to `setuptools.build_meta:__legacy__.build_wheel` failed (exit status: 1)
[stderr]
/Users/rene/.cache/uv/builds-v0/.tmp5o48Gd/lib/python3.13/site-packages/setuptools/dist.py:759: SetuptoolsDeprecationWarning: License classifiers are deprecated.
!!
********************************************************************************
Please consider removing the following classifiers in favor of a SPDX license expression:
License :: OSI Approved :: Python Software Foundation License
See https://packaging.python.org/en/latest/guides/writing-pyproject-toml/#license for details.
********************************************************************************
!!
self._finalize_license_expression()
bitarray/_bitarray.c:131:23: error: expression is not assignable
131 | Py_SIZE(self) = newsize;
| ~~~~~~~~~~~~~ ^
bitarray/_bitarray.c:156:19: error: expression is not assignable
156 | Py_SIZE(self) = newsize;
| ~~~~~~~~~~~~~ ^
bitarray/_bitarray.c:177:18: error: expression is not assignable
177 | Py_SIZE(obj) = nbytes;
| ~~~~~~~~~~~~ ^
bitarray/_bitarray.c:1372:14: error: call to undeclared function 'PyEval_CallObject'; ISO C99 and later do not support implicit function declarations
[-Wimplicit-function-declaration]
1372 | result = PyEval_CallObject(reader, rargs);
| ^
bitarray/_bitarray.c:1372:12: error: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
1372 | result = PyEval_CallObject(reader, rargs);
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bitarray/_bitarray.c:1511:14: error: call to undeclared function 'PyEval_CallObject'; ISO C99 and later do not support implicit function declarations
[-Wimplicit-function-declaration]
1511 | result = PyEval_CallObject(writer, args);
| ^
bitarray/_bitarray.c:1511:12: error: incompatible integer to pointer conversion assigning to 'PyObject *' (aka 'struct _object *') from 'int' [-Wint-conversion]
1511 | result = PyEval_CallObject(writer, args);
| ^ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
bitarray/_bitarray.c:3206:28: error: expression is not assignable
3206 | Py_TYPE(&Bitarraytype) = &PyType_Type;
| ~~~~~~~~~~~~~~~~~~~~~~ ^
bitarray/_bitarray.c:3207:31: error: expression is not assignable
3207 | Py_TYPE(&SearchIter_Type) = &PyType_Type;
| ~~~~~~~~~~~~~~~~~~~~~~~~~ ^
bitarray/_bitarray.c:3208:31: error: expression is not assignable
3208 | Py_TYPE(&DecodeIter_Type) = &PyType_Type;
| ~~~~~~~~~~~~~~~~~~~~~~~~~ ^
bitarray/_bitarray.c:3209:33: error: expression is not assignable
3209 | Py_TYPE(&BitarrayIter_Type) = &PyType_Type;
| ~~~~~~~~~~~~~~~~~~~~~~~~~~~ ^
11 errors generated.
error: command '/usr/bin/cc' failed with exit code 1
hint: This usually indicates a problem with the package or the build environment.
help: `bitarray` (v0.9.3) was included because `scancode-toolkit` (v21.2.9) depends on `bitarray`
bitarray v0.9.3 is 6 years old: https://pypi.org/project/bitarray/#history
I can only find outdated test data: https://github.com/aboutcode-org/scancode-toolkit/blob/develop/tests/packagedcode/data/pypi/develop/scancode_toolkit.egg-info/requires.txt#L3
bitarray<1.0.0,>=0.8.1
But of course nobody should use requirements pinning anymore, exactly why packages can't be installed on newer Python versions anymore when updates are forbidden. https://iscinumpy.dev/post/bound-version-constraints/
help:
bitarray(v0.9.3) was included becausescancode-toolkit(v21.2.9) depends onbitarray
The latest version of scancode-toolkit is 32.4.1 which should support Python 3.13. You are trying to install 21.2.9 from over four years ago. It looks like you are having a separate issue and should create a new ticket for that.
But of course nobody should use requirements pinning anymore, exactly why packages can't be installed on newer Python versions anymore when updates are forbidden.
I was referring to the fact that tests are all run in upstream CI using library versions pinned in requirements.txt, which is generally a good practice, provided that they are updated on a regular basis and not pinned to versions that are significantly out of date.
Pinning bitarray<1.0.0,>=0.8.1 in the packaging metadata probably made sense at the time if it was expected that bitarray 1.0.0 would have breaking changes.
You can see my command line, that's what uvx resolves to when ommitting the version. But
uvx scancode-toolkit==32.4.1
isn't much better for Python 3.13.
× No solution found when resolving tool dependencies:
╰─▶ Because only the following versions of typecode-libmagic are available:
typecode-libmagic<=5.39.210223
typecode-libmagic==5.39.210531
and typecode-libmagic>=5.39.210223 has no wheels with a matching platform tag (e.g., `macosx_15_0_arm64`), we can conclude that typecode-libmagic>=5.39.210223 cannot be
used. (1)
And because typecode[full]>=30.0.1 depends on typecode-libmagic>=5.39.210223 and only the following versions of typecode[full] are available:
typecode[full]<=30.0.1
typecode[full]==30.0.2
we can conclude that typecode[full]>=30.0.1,<30.0.2 cannot be used. (2)
Because we know from (1) that typecode-libmagic>=5.39.210223 cannot be used and typecode[full]==30.0.2 depends on typecode-libmagic>=5.39.210223, we can conclude that
typecode[full]==30.0.2 cannot be used.
And because we know from (2) that typecode[full]>=30.0.1,<30.0.2 cannot be used, we can conclude that typecode[full]>=30.0.1 cannot be used.
And because scancode-toolkit==32.4.1 depends on typecode[full]>=30.0.1 and you require scancode-toolkit==32.4.1, we can conclude that your requirements are unsatisfiable.
hint: Wheels are available for `typecode-libmagic` (v5.39.210531) on the following platforms: `manylinux1_x86_64`, `macosx_10_14_x86_64`, `win_amd64`
But I didn't want to steal your thread, didn't know that you don't support arm64 or Apple Silicon yet, typecode-libmagic doesn't seem to be maintained anymore, last release is 4 years old: https://pypi.org/project/typecode-libmagic/#files.
I can only recommend using the standard cibuildwheel action if possible to build for all platforms and architectures. https://github.com/pypa/cibuildwheel#example-setup
But I didn't want to steal your thread, didn't know that you don't support arm64 or Apple Silicon yet, typecode-libmagic doesn't seem to be maintained anymore, last release is 4 years old
This is a separate issue and is tracked in #3958.
@gotmax23 Thanks for all this. This is awesome! As for pkginfo2, it needs to be updated, but the reason for the fork and patch is security: pkginfo does insecure evaluation of code that is possibly dangerous when processing random code at scale. It may be OK for pkginfor, but is not OK for ScanCode
