google
[infra] Upgrade Python to 3.10.14 in base-builder & base-runner Images
[!NOTE]
I was looking for somewhere to get feedback from maintainers about this approach to the Python 3.10 upgrade before attempting it, but the discussion surrounding a Python upgrade has been rather fragmented across many issues, PRs, and comment chains.For that reason, I felt it would be easier to propose with a working example and dedicated PR.
Fixes:
- #11419
- #9638
Supersedes:
- #9532
- #11420
Changes
The changes introduced here upgrade Python from 3.8 to 3.10.14 inside the base-builder and base-runner images.
Base Image Changes
| Image | Before Changes | After Changes |
|---|---|---|
| base-builder | Compiled Python 3.8 from source using official release servers at https://www.python.org/ftp/python/. | Compiles Python 3.10.14 (the latest 3.10 release) from source using official release servers at https://www.python.org/ftp/python/. |
| base-runner | Installed Python 3.8 from the default apt repository provided by the Ubuntu 20.04 image. | Uses a multi-stage build to copy the Python 3.10.14 interpreter compiled by the base-builder image, ensuring version sync and saving build time by re-using a pre-built version. |
Known Impact on Projects
3.9 Workarounds That Can Be Removed
| Project | Fix Link |
|---|---|
| dask | https://github.com/DaveLak/oss-fuzz/commit/417bbf566e3ffd962e09a50903282233b3676b69 |
| docutils | https://github.com/DaveLak/oss-fuzz/commit/e4c21ffbd8baa83d97d7dd0e1f8e3e7adeae43d1 |
| dovecot | https://github.com/DaveLak/oss-fuzz/commit/7ab3ab6e2604396f99398a1236ff6ddb91b04430 |
| nbclassic | https://github.com/DaveLak/oss-fuzz/commit/5509b4e221ecad6136b353780daf1ceb3f457f53 |
| pandas | https://github.com/DaveLak/oss-fuzz/commit/0642a7afa96d193439cc9ebd7659a0fccc7e5c1b |
| pybind11 | https://github.com/DaveLak/oss-fuzz/commit/a5bbdb3dfbddd7f37dd7737cb57bb39c9ec5cc5f |
| pyodbc | https://github.com/DaveLak/oss-fuzz/commit/afa2b5edc96d581329fe0405d463b026abbea3ad |
| qpid-proton | https://github.com/DaveLak/oss-fuzz/commit/f5bf75628649fea5696fd8e95e1e698092ea4e74 |
Anticipated Build Failures
Preexisting Failures
Fix is Prepared
| Project | Fix Link |
|---|---|
| airflow | https://github.com/DaveLak/oss-fuzz/commit/60a03686280b63e88773ef9b25cd191ab8e959d0 |
| ipython | https://github.com/DaveLak/oss-fuzz/commit/21ac68e9ba44e063d04b3df394330f6490081bcb |
| networkx | https://github.com/DaveLak/oss-fuzz/commit/fc2f8c5481262849b586ebb8afdfdf3e6672f44f |
| numpy | https://github.com/DaveLak/oss-fuzz/commit/9383c876dccad016f169d43a948c5bdc45b2894d |
| tensorflow-addons | https://github.com/DaveLak/oss-fuzz/commit/eed2bea1190abd51f3c621ccc923fecb6585a67c |
| django (coverage build) | https://github.com/DaveLak/oss-fuzz/commit/c724d61eae52e309726dc0d90f22bdbbecee04e7 |
| proto-plus-python | https://github.com/DaveLak/oss-fuzz/commit/37d973ee8726c33678da37fb41e48d264418f097 |
| dnspython | The upgraded pip version in the base-builder fixes the currently failing build. |
Fix Requires Upstream Changes
| Project | Issue |
|---|---|
| pyvex | Currently failing on python 3.9 because archinfo dependency requires >=3.10. Fails after the 3.10 upgrade because the upstream build script needs python3.9 replaced with python3. |
Requires More Investigation
| Project | Issue |
|---|---|
| matplotlib | Upgrading Python & Pyinstaller does resolve the build issues, but an error in the fuzz harness is exposed and must be resolved for check_build to pass. The exception: TypeError: Parser.non_math() takes 2 positional arguments but 4 were given" in "File "fuzz_plt.py", line 43, in TestOneInput. |
| scipy | Upgrading Python & Pyinstaller does resolve the build issues, but an error in the build step causes the build to fail. The error seems related to the linking: "/usr/bin/ld: /usr/bin/ld: DWARF error: invalid or unhandled FORM value: 0x25". When export LDFLAGS="-fuse-ld=lld" is set, the error becomes: "ld.lld: error: undefined symbol: __asan_report_store4". |
| pandas (Introspector only) | This workaround in build.sh is the issue. |
| pycrypto | Failing with error: "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats". Seems like the issue described here. Pycrypto is deprecated and this is unlikely to be fixed upstream. |
Possible Future Improvements
Using the base-builder image in a multi-stage build to copy the pre- compiled Python into base-runner is effective, but feels like a workaround that may be introducing tech debt. A cleaner approach would be to extract the Python compilation into a discrete base image similar to how base-clang works, and use that as the multi-stage builder in images that need it.
Fuzz Introspector Caveat
Fuzz Introspector currently uses Python 3.9. While an upgrade to 3.10 is not expected to introduce any new issues, it was not tested on these changes and may require additional work.
Motivation
- Python 3.8 is reaching end of life in October 2024.
- The Scientific Python Community already encourages dropping 3.8 support.
- This is evident when looking at which projects have resorted to upgrading to newer Pythons using ad-hoc workarounds (see
numpy,scipy,pandas, etc.)
- This is evident when looking at which projects have resorted to upgrading to newer Pythons using ad-hoc workarounds (see
- It is likely that more Python projects will begin dropping support for 3.8, further increasing the number of broken builds and ad-hoc workarounds.
- Code coverage does not work on Python projects that use Python 3.10+ syntax.
- Previous attempts at upgrading Python have stalled (see https://github.com/google/clusterfuzz/issues/3290#issuecomment-1699573706 & the issues linked under "Supersedes" above.)
- In recognition of the fact that OSS-Fuzz maintainers are stretched thin, I thought I'd give it a shot.
/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer
/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer
@jonathanmetzman I think e1a6e9fe59e29563f0a5dfbaeecd9d8c71762fdd should fix the broken coverage builds.
/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer
Step #1: INFO:root:----------------------------Build result---------------------------- Step #1: INFO:root:Trial build end time: 2024-07-01 23:47:03.076958 Step #1: INFO:root:Failed project, Statuses, Logs Step #1: INFO:root:adal, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-59e9438d-1ff0-4ffa-8224-8111645a98af.txt Step #1: INFO:root:antlr3-java, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0433cbe9-0538-439b-a4d4-dbe4f848e62f.txt Step #1: INFO:root:antlr4-java, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-acc02e30-e5ac-4799-a338-6b39bbbb0344.txt Step #1: INFO:root:apache-commons-bcel, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5bd6cbdc-bf98-4e8f-9b7c-696c93692f68.txt Step #1: INFO:root:apache-commons-configuration, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-385d8181-ff5c-4bd1-8737-cc2c18942098.txt Step #1: INFO:root:apache-commons-io, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-62d9a1a7-b2dd-4630-84c2-ce547d42b242.txt Step #1: INFO:root:args4j, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-6a2d6890-f82b-44e7-9974-c7c022d22235.txt Step #1: INFO:root:augeas, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-68014ad0-7f36-4c36-b6a7-7727142ecb5d.txt Step #1: INFO:root:black, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-cd576622-e346-4e48-ac1a-5d584253a818.txt Step #1: INFO:root:capstone, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f4bc5b49-1878-4646-8427-1188c9e404c8.txt Step #1: INFO:root:capstone, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ece1df41-1086-4071-9782-5f01c06446a9.txt Step #1: INFO:root:charset_normalizer, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-80b23fcb-6554-4247-bbd8-4daf9f1b75d5.txt Step #1: INFO:root:connexion, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e8a74cea-bb59-47a1-844b-820bd384fb00.txt Step #1: INFO:root:cryptography, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0d10efa6-1e38-49f9-b617-91b11f6da301.txt Step #1: INFO:root:filesystem_spec, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5f73e2aa-1b04-4de6-9fe9-442d936b6d97.txt Step #1: INFO:root:flask-restx, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-4a9d8672-dc50-4dce-a231-cfcf1b82ba03.txt Step #1: INFO:root:fwupd, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-65397082-aaf0-434d-9475-5a75186e9e74.txt Step #1: INFO:root:fwupd, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1bd67925-c598-4ef4-ace1-b2f29ceaa747.txt Step #1: INFO:root:g-api-pubsub, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f8832604-6ab1-403f-bb90-2f96a6e7e505.txt Step #1: INFO:root:g-api-py-oauthlib, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-c704ea64-418d-4a89-b173-1cbd521ce444.txt Step #1: INFO:root:g-api-python-bigquery-storage, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7eea83fa-516f-404a-8968-ebb6f2c3f70a.txt Step #1: INFO:root:g-api-python-cloud-core, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3cab03b6-ae60-4c00-8527-902ed1ee6d90.txt Step #1: INFO:root:g-api-python-firestore, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e1fb8d0c-eee8-41c7-aec4-17c3e92151fa.txt Step #1: INFO:root:g-api-python-tasks, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-a396dbc7-5019-4cf4-9cec-3442a9d5bb32.txt Step #1: INFO:root:g-api-resource-manager, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-c9bf8e2b-8746-4900-b9d0-c09422b979e2.txt Step #1: INFO:root:g-api-secret-manager, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-aafc7ad3-65ec-4d6c-b736-05e795bc939c.txt Step #1: INFO:root:g-cloud-logging-py, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5e55f45d-7a25-40ed-9f2b-40834a82406c.txt Step #1: INFO:root:g-http-java-client, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-355832cb-00bc-40fd-a994-31f7268f08bd.txt Step #1: INFO:root:g-py-bigquery, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-69f63c0d-d1b1-4b1c-a21d-4f7fdd77e679.txt Step #1: INFO:root:gc-iam, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5aca40d1-a633-4ae1-baa1-1081cefdc067.txt Step #1: INFO:root:gcloud-error-py, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-523d22bc-41e0-424a-9ed8-56715e56998e.txt Step #1: INFO:root:gcp-python-cloud-storage, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2dc76ef5-0ab5-49f6-90b1-f29bc1ddc4bf.txt Step #1: INFO:root:h5py, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-722b2644-4760-42c3-9850-d6d8c2791fec.txt Step #1: INFO:root:hadoop, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-dea4441f-e119-4172-8a82-e58a5b0c8174.txt Step #1: INFO:root:httplib2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d0b67e02-e149-4ca6-9cca-d3070f687446.txt Step #1: INFO:root:httpretty, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-73678cae-3e84-4112-89eb-b35376c28327.txt Step #1: INFO:root:ijson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ac4359f7-49a4-4b36-be70-84a418eb6116.txt Step #1: INFO:root:ipykernel, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-bfa3c47a-9c0c-497d-b518-e3303ab29a82.txt Step #1: INFO:root:isodate, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-be3ab31f-f67b-4c3b-af35-2b45cb73d873.txt Step #1: INFO:root:jedi, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0d6209c9-1078-41ff-bfae-d2ab2964c791.txt Step #1: INFO:root:jupyter_server, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-490864fa-ccd8-49f8-bc01-cb51a7b2bb7c.txt Step #1: INFO:root:kie-soup, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-863ff132-84b4-4db3-bfdb-ba18c3629c5b.txt Step #1: INFO:root:knot-dns, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d7b332d7-011e-4bde-89c4-8b5f83f5f052.txt Step #1: INFO:root:libidn, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f2cb8817-7a0b-46ab-9241-fc35d4c9ad21.txt Step #1: INFO:root:libidn2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3cf3ca45-dd6d-4d69-b649-ba3429c637ba.txt Step #1: INFO:root:libpsl, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-3fd2f1aa-74e8-4ac0-86c4-26713a0163f8.txt Step #1: INFO:root:libtasn1, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e84497e0-96d7-41da-a6ab-9fd27257e39e.txt Step #1: INFO:root:libxls, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-b615294d-15c4-47de-9873-f3a16f3bcf54.txt Step #1: INFO:root:libxls, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0cb093ce-d7f1-4f50-be29-8d6d01bbb100.txt Step #1: INFO:root:libxml2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2ee82ccc-0618-410a-af96-5d6138762889.txt Step #1: INFO:root:looker-sdk, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-8f4b474f-514e-4c85-9f28-522aacc6469f.txt Step #1: INFO:oauth2client.transport:Refreshing due to a 401 (attempt 1/2) Step #1: INFO:root:msal, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-8acd07c4-6ffe-4d90-a6c3-676590175d08.txt Step #1: INFO:root:nbclassic, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1eb60faa-2968-4ee5-9834-73cc3e93cc7e.txt Step #1: INFO:root:nbclassic, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d0ab947b-d5f0-445b-93b0-3d793dc02cca.txt Step #1: INFO:root:nbformat, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-21a0e098-a263-4820-80fa-0438f9da2470.txt Step #1: INFO:root:ntlm2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7434edaf-572f-4f89-9446-d0e670996ade.txt Step #1: INFO:root:orjson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f76408e1-fbc1-46b7-a36a-1ca528bc55d3.txt Step #1: INFO:root:orjson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5bd98fe6-5b20-4c73-97a1-56e0bb9902aa.txt Step #1: INFO:root:paramiko, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-dc135c35-a9e7-4b35-8d5d-b672feafa2fe.txt Step #1: INFO:root:parso, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5ce6388d-5599-40d9-ab0a-d8cb7611b732.txt Step #1: INFO:root:pasta, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-7ecf9b62-0452-4eb4-a950-0faaa57a1af1.txt Step #1: INFO:root:pffft, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2d82a68e-2374-4dbb-886a-d8c3253bd34d.txt Step #1: INFO:root:pffft, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-cb3bbe5f-437e-41f6-927d-cc1bec8ccc10.txt Step #1: INFO:root:pillow, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e19625fb-a984-47b1-be32-f92b4c76fc7f.txt Step #1: INFO:root:pillow, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-80ededd3-8c64-45c5-9913-6447e6f734e6.txt Step #1: INFO:root:proto-plus-python, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-6868d820-88c1-4fe8-a7e4-054e86554da7.txt Step #1: INFO:root:pyasn1-modules, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-df5d0051-08d2-4f08-b02c-314da42fea63.txt Step #1: INFO:root:pybind11, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e5f9f0f2-d1d8-4a2c-bf41-4c89232e3cec.txt Step #1: INFO:root:pybind11, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-380ecce4-400f-42ff-8d58-5ee5e8cb90ca.txt Step #1: INFO:root:pycrypto, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f655175f-cd8e-4f76-abb5-d4092e72c715.txt Step #1: INFO:root:pyparsing, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-829f2448-5777-44a8-9f20-70dd21c7e7d5.txt Step #1: INFO:root:python-pypdf, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-263bb117-8446-4b64-8a18-ce0c17e87db1.txt Step #1: INFO:root:pyxdg, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ee6476d9-3fe2-487f-8d78-529ea375a416.txt Step #1: INFO:root:pyzmq, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d8550a00-608c-46fa-9210-cb6c33ed4d86.txt Step #1: INFO:root:sacremoses, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-611d1c7e-f36f-4bf7-9c62-59950b039685.txt Step #1: INFO:root:simplejson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-4eb149c7-8afc-48a1-a757-2916f02aa7db.txt Step #1: INFO:root:six, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1d47f866-910f-4b22-b220-6f354e445360.txt Step #1: INFO:root:soupsieve, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-bf945104-19ff-4319-bf98-e553bdde0f66.txt Step #1: INFO:root:tablesaw, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-4ef78f17-4fe6-446e-90a2-66c40033c153.txt Step #1: INFO:root:tinyusb, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1a6f010e-cf64-4d63-a1e3-7ca9a97c168f.txt Step #1: INFO:root:toolbelt, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2642efe6-d83b-4817-ae9a-f241358e7141.txt Step #1: INFO:root:wget2, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-a1c02746-8f70-472d-85b3-31f56bb520b4.txt Step #1: INFO:root:xlrd, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-2a2bad59-7dd1-44f7-8ab9-f65b9487a2b6.txt Step #1: INFO:root:xmltodict, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-085add0f-851d-495a-8389-60b9bf7fc449.txt Step #1: INFO:root:yamlbeans, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-857d16ce-6d90-4e3f-bd95-13e644a8855c.txt Step #1: INFO:root:yarl, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-efd06c32-7445-4fed-a470-82b9f33afae2.txt Step #1: INFO:root:django, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-b92e8311-f2dc-412f-83fa-601d54c74c47.txt Step #1: INFO:root:python-markdown, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-1b32acfa-f5be-4220-880c-10a1b67c5074.txt
Thanks for grabbing that list @DonggeLiu! I've started looking over them and will report back.
The few I've seen so far appear to be caused by ad-hock Python version updated in build scripts which is reassuring.
FYI for anyone interested:
Here's the list formatted as table:
Failures Table
| Project | Status | Log URL |
|---|---|---|
| adal | FAILURE | log |
| antlr3-java | FAILURE | log |
| antlr4-java | FAILURE | log |
| apache-commons-bcel | FAILURE | log |
| apache-commons-configuration | FAILURE | log |
| apache-commons-io | FAILURE | log |
| args4j | FAILURE | log |
| augeas | FAILURE | log |
| black | FAILURE | log |
| capstone | FAILURE | log |
| capstone | FAILURE | log |
| charset_normalizer | FAILURE | log |
| connexion | FAILURE | log |
| cryptography | FAILURE | log |
| filesystem_spec | FAILURE | log |
| flask-restx | FAILURE | log |
| fwupd | FAILURE | log |
| fwupd | FAILURE | log |
| g-api-pubsub | FAILURE | log |
| g-api-py-oauthlib | FAILURE | log |
| g-api-python-bigquery-storage | FAILURE | log |
| g-api-python-cloud-core | FAILURE | log |
| g-api-python-firestore | FAILURE | log |
| g-api-python-tasks | FAILURE | log |
| g-api-resource-manager | FAILURE | log |
| g-api-secret-manager | FAILURE | log |
| g-cloud-logging-py | FAILURE | log |
| g-http-java-client | FAILURE | log |
| g-py-bigquery | FAILURE | log |
| gc-iam | FAILURE | log |
| gcloud-error-py | FAILURE | log |
| gcp-python-cloud-storage | FAILURE | log |
| h5py | FAILURE | log |
| hadoop | FAILURE | log |
| httplib2 | FAILURE | log |
| httpretty | FAILURE | log |
| ijson | FAILURE | log |
| ipykernel | FAILURE | log |
| isodate | FAILURE | log |
| jedi | FAILURE | log |
| jupyter_server | FAILURE | log |
| kie-soup | FAILURE | log |
| knot-dns | FAILURE | log |
| libidn | FAILURE | log |
| libidn2 | FAILURE | log |
| libpsl | FAILURE | log |
| libtasn1 | FAILURE | log |
| libxls | FAILURE | log |
| libxls | FAILURE | log |
| libxml2 | FAILURE | log |
| looker-sdk | FAILURE | log |
| msal | FAILURE | log |
| nbclassic | FAILURE | log |
| nbclassic | FAILURE | log |
| nbformat | FAILURE | log |
| ntlm2 | FAILURE | log |
| orjson | FAILURE | log |
| orjson | FAILURE | log |
| paramiko | FAILURE | log |
| parso | FAILURE | log |
| pasta | FAILURE | log |
| pffft | FAILURE | log |
| pffft | FAILURE | log |
| pillow | FAILURE | log |
| pillow | FAILURE | log |
| proto-plus-python | FAILURE | log |
| pyasn1-modules | FAILURE | log |
| pybind11 | FAILURE | log |
| pybind11 | FAILURE | log |
| pycrypto | FAILURE | log |
| pyparsing | FAILURE | log |
| python-pypdf | FAILURE | log |
| pyxdg | FAILURE | log |
| pyzmq | FAILURE | log |
| sacremoses | FAILURE | log |
| simplejson | FAILURE | log |
| six | FAILURE | log |
| soupsieve | FAILURE | log |
| tablesaw | FAILURE | log |
| tinyusb | FAILURE | log |
| toolbelt | FAILURE | log |
| wget2 | FAILURE | log |
| xlrd | FAILURE | log |
| xmltodict | FAILURE | log |
| yamlbeans | FAILURE | log |
| yarl | FAILURE | log |
| django | FAILURE | log |
| python-markdown | FAILURE | log |
@jonathanmetzman @DonggeLiu Could you please start a trial build with the latest changes?
Thanks in advance!
The following information provides context for the changes and is intended for reference.
Analysis of Recent Failures
After reviewing the recent run failures, here's what I found:
-
PySecSan Integration: Several projects that failed have
pyscansanitizer enabled (ENABLE_PYSECSAN="1"). The resolution should be achieved with commit 3e3fa52a81cfe1e459f2430c4bce98fb8ab23834 -
Dependency Resolution: Many failures were due to issues with dependency resolution caused by outdated Pyinstaller and Setuptools versions, which are resolved by commit e6fc52c3da5801b87d0b7b0950a2880db3aeb13b. Initially, I wasn’t going to attempt a version bump of those deps, but seeing as how those very old dependency versions are a common source of broken builds in both Python 3.8 & 3.10, it's apparent that the bump is necessary.
-
Pillow will continue to fail but a patch is already prepared via #12326
JVM/Java project falures
The failures in JVM/Java projects appear to be network errors unrelated to this PR. It seems that repo.maven.apache.org might be dropping requests due to excessive download attempts from the OSS-Fuzz infrastructure. The following error is common in the JVM project logs:
[ERROR] Failed to execute goal on project antlr3-maven-plugin:
Could not resolve dependencies for project
org.antlr:antlr3-maven-plugin:maven-plugin:3.5.4-SNAPSHOT:
Could not transfer artifact
org.apache.maven:maven-artifact:jar:3.8.4 from/to central
(https://repo.maven.apache.org/maven2): transfer failed for
https://repo.maven.apache.org/maven2/org/apache/maven/maven-artifact/3.8.4/maven-artifact-3.8.4.jar:
Connection reset -> [Help 1]
Edit: Disregard the comment below. Rerunning it on a merge was successful.
~The CI check that is failing appears to be due to a transient network request failure and unrelated to the changes in this PR.~ From the logs:
fatal: unable to access 'https://github.com/llvm/llvm-project.git/': Failed to connect to github.com port 443: Connection timed out
@jonathanmetzman, could you trigger another trial build whenever you have a chance? Thanks!
/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer
Pillow will continue to fail but a patch is already prepared via https://github.com/google/oss-fuzz/pull/12326
That PR has now been merged into master.
Thanks, @DonggeLiu.
I've looked into the latest trial build failures from the build at 9db721e90d463a64cb7d890fdfd019302c3352c8 (GH Run: https://github.com/google/oss-fuzz/runs/29280838754) and outlined my findings below.
Latest trial build failures
Fix Identified
I've identified fixes for the failed projects in this section and committed them to a new branch on my fork for now.
Project: configparser
- Log URL: log
- Reason for Failure: Does not work with `setuptools>=70'.
- Fix: https://github.com/DaveLak/oss-fuzz/commit/c44388e6974e767251cba17a4cf4590be44bc966
Project: fwupd
- Log URL: log
- Reason for Failure: Missing Python package dependency.
- Fix: https://github.com/DaveLak/oss-fuzz/commit/d04205e8f0762821465f6217921b196c20c66940
Project: gitpython
- Log URL: log
- Reason for Failure: The fuzzing harness that failed is flaky.
- Fix: I maintain that integration and will fix it. This is not a blocker for the PR.
Project: nbclassic
- Log URL: log
- Reason for Failure: Hardcoded references to Python 3.8 installation paths in build script.
- Fix: https://github.com/DaveLak/oss-fuzz/commit/b338e1ffe1e92e24b3b1d9d2544096a4db71aed1
Project: pffft
- Log URL: log
- Reason for Failure:
pffft/generate_seed_corpus.pyhelper script dependencynumpywas installed viaapt-getbut should use pip instead. - Fix: https://github.com/DaveLak/oss-fuzz/commit/708a18e5ba3089d918d3d89c099e525f6030ba29
Project: proto-plus-python
- Log URL: log
- Reason for Failure: Python syntax
- Fix: https://github.com/DaveLak/oss-fuzz/commit/8a4f8e21fd73f81d4a78d7d9b147e8e122814f93
Project: pybind11
- Log URL: log
- Reason for Failure: Hardcoded references to Python 3.8 installation paths in build script.
- Fix: https://github.com/DaveLak/oss-fuzz/commit/2633c0289968c094b52ad8fa665e6858238f6d0b
Project: pyzmq
- Log URL: log
- Reason for Failure: Missing
libzmq3-devapt-get dependency. - Fix: https://github.com/DaveLak/oss-fuzz/commit/41b04bf77ade9c3fe5a02de50073a106d72d3205
Project: django
- Log URL: log
- Reason for Failure: An ad-hoc workaround using the Dead Snakes PPA to install Python 3.10
- Fix: https://github.com/DaveLak/oss-fuzz/commit/185ac6bdda1eb9872f4625f4d0237837a2daf423
Failure Related but Require Further Investigation
Project: six
- Log URL: log
- Reason for Failure: TBD
- Fix: TBD
Project: ijson
- Log URL: log
- Reason for Failure: TBD
- Fix: N/A
ijson appears to be the only coverage failure in the latest trial build run.
Project: pycrypto
- Log URL: log
As noted in the PR description above, Pycrypto is deprecated and this is unlikely to be fixed upstream.
The latest failure is different, but even if it is fixes, it will continue failing with the error: "SystemError: PY_SSIZE_T_CLEAN macro must be defined for '#' formats". It seems like the issue described here.
Likely Unrelated?
The project below failed with the following message:
docker: Error response from daemon: image with reference gcr.io/oss-fuzz-base/base-runner-testing-upgrade-python-to-3-10-14 was found but does not match the specified platform: wanted linux/arm64, actual: linux/amd64.
I was not able to reproduce this locally.
Project: lua
- Log URL: log
Project: libxml2
- Log URL: log
Project: tinyusb
- Log URL: log
Project: tarantool
- Log URL: log
Unrelated Failures
Project: bloaty
- Log URL: log
- Reason for Failure: ; Build started failing on 8/27/2024 with the same error logged
- Fix: N/A
I'm not sure what these failed "Project tests / build" CI checks are about, but it seems to happen intermittently on this PR and the logs make me think it has something to do with a stale Docker image cache?
@DonggeLiu or @jonathanmetzman, could either of you comment on what next steps are necessary to get this landed?
Given the latest trial build is showing the changes proposed in this PR will only break ~9 project builds (7 of which I've prepared a patch for) and the significantly larger number projects that would benefit from the Python version bump, I'm eager to bring this over the finish line.
Is there anything more I can do for the time being?
@DonggeLiu @jonathanmetzman Just bumping this up in case it got missed. Could you share any thoughts on the feasibility of this Python upgrade approach and whether it’s worth keeping the branch updated?
Fyi any project that depends on zephyr is likely broken until this lands see https://github.com/zephyrproject-rtos/zephyr/commit/9d1b36126535c611617c5e8edcf06d70f5879287
Apologies for the very late reply @DaveLak, and thank you for your work here.
@DonggeLiu or @jonathanmetzman, could either of you comment on what next steps are necessary to get this landed?
Given the latest trial build is showing the changes proposed in this PR will only break ~9 project builds (7 of which I've prepared a patch for) and the significantly larger number projects that would benefit from the Python version bump, I'm eager to bring this over the finish line.
Do you have links to / PRs for these patches? Our main concern with any kind of upgrade is causing new build failures for existing projects.
Hello - thank you for working on this. I found this issue because fuzzing for websockets stopped working when I dropped support for Python 3.8, which is now EOL.
I see that you've been paying attention to projects that will be broken when you merge this change. Have in mind that some projects (starting with mine) will be fixed :-)
Our main concern with any kind of upgrade is causing new build failures for existing projects.
I hear you; however, you're probably reaching the point where you're causing more failures by sticking to an end-of-life Python than you'd cause by upgrading!
Do you have links to / PRs for these patches?
There's here: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556
/gcbrun trial_build.py all --sanitizer coverage address --fuzzing-engine libfuzzer
Apologies for the very late reply @DaveLak
Not a problem. I understand there are many reasons for that. I'm grateful for you taking the time to consider it now 🙂 Thanks, @oliverchang!
Do you have links to / PRs for these patches?
I have a branch in my fork that I based on this PR branch with each commit containing a patch scoped to an individual project: https://github.com/DaveLak/oss-fuzz/tree/infra/upgrade-python-to-3-10-14/patch-failing-projects
The commit message for each should describe the reason for each failure and the fix. I also replied to the previous trial build with the same info here https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556
The PR description has links to several similarly organized patches from failures I saw when I opened this PR, but some of those might be stale at this point as I based them on testing from ~5 months ago. We'll see what the trial build says I guess.
[!NOTE]
Please Note: I am not sure what the best method for contributing these patches would be. All of the patches I am referring to are for the OSS-Fuzz repo but if I merge the commits into this PR branch I believe the "PR must be approved by known contributors before it can be merged" bot will have a fit.
Our main concern with any kind of upgrade is causing new build failures for existing projects.
Of course; mine too 🙂
I spent quite a bit of time reviewing failures locally and in the trial builds. As of the last trial build, I have a patch prepared for all but 3 (with at least 1, pycrypto, being deprecated and unlikely to be fixed upstream.)
Meanwhile, Python 3.8 has reached EOL and a significant portion of OSS-Fuzz integrated Python projects no longer support it. Most notably, the Scientific Python Community no longer requests member projects to support 3.8, and actually has dropped support for 3.10 as well but I still think this PR is worth trying to merge. It will fix many currently broken projects and make an upgrade to >=3.11 easier to manage afterwards.
This doesn't seem to cause any problems in cifuzz: https://github.com/jonathanmetzman/cifuzz-example/actions/runs/11918843132/job/33217203025?pr=11
It does break ~10 python builds. 2 seem very easy to fix but I'm not sure about the others. Would you have time to look into these? ijson, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-5030ab50-e0ce-45da-bd40-ce748e5db149.txt mrab-regex, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-8ba9c1e4-3f05-4d3c-b3de-272ef4030d34.txt nbclassic, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-d0e601ef-f70f-43e1-afbb-19febde4fd4b.txt - COPY python3.10 instead of 3.8 pffft, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-32b99100-6ebc-4f62-9a84-150355fa0718.txt - Install numpy proto-plus-python, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-6830663c-d194-4c8e-b3d9-4950e3767ae6.txt pybind11, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-f3dd7860-d923-4600-be05-c6f21c5d5fdf.txt pybind11, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-ae58fd24-a9fa-4186-9adc-4ff992e328eb.txt pycrypto, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-940fdbca-2991-4bf4-94e0-293b36a8eee9.txt pyzmq, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-0aa16e95-c5b9-4e1b-ac38-805aea6e52ee.txt six, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-b78a537e-5446-4c48-98f8-3f26d66abf49.txt django, FAILURE, https://oss-fuzz-gcb-logs.storage.googleapis.com/log-e8c2228f-87fe-46d8-8cb4-4c7d7296df4b.txt
It does break ~10 python builds. 2 seem very easy to fix but I'm not sure about the others. Would you have time to look into these?
Sure, thanks for the links @jonathanmetzman. I took a quick look and shared my findings in the table below.
TL;DR
These are mostly the same. I have a patch prepared for all but 2 (see https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556). Of the two without a patch prepared:
- One (mrab-regex) is failing to build on
master. - The other (pycrypto) does not support Python 3.10 and the project is deprecated so this is unlikely to be fixed upstream.
Here's more details based on a quick review:
| Project | Is New Since 2024-08-26? | Is Patch Already Prepared? | Notes |
|---|---|---|---|
| ijson | ❌ | ✅ | Only coverage build failing |
| mrab-regex | ✅ | ❌ | Only coverage build failing. This might be due to a lack of corpora since the project is failing to build on master? |
| nbclassic | ❌ | ✅ | Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. Patch is already prepared. |
| pffft | ❌ | ✅ | Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. Patch is already prepared. |
| proto-plus-python | ❌ | ✅ | Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. Patch is already prepared. |
| pybind11 | ❌ | TBD | (I think?) Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. Patch is already prepared. |
| pycrypto | ❌ | ❌ | Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. The issue is due to upstream's use of removed Python features & the project is deprecated and unmaintained so this is unlikely to be fixed. |
| pyzmq | ❌ | ✅ | Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. Patch is already prepared. |
| six | ❌ | ✅ | @radarhere contributed a patch to my fork; see that PR for details: https://github.com/DaveLak/oss-fuzz/pull/1 |
| django | ❌ | ✅ | Same issue as I mentioned previously: https://github.com/google/oss-fuzz/pull/12027#issuecomment-2314118556. Patch is already prepared. |
On a side-note: Is there an easy way I can pull the pre-built Docker images from this PR locally to test with? Rebuilding the base images locally is the most time consuming part of testing.
Thanks @DaveLak ! Would you mind including the patches you already have for the failing projects in the same PR?
Would you mind including the patches you already have for the failing projects in the same PR?
All set, @oliverchang!
The build (libfuzzer, address, i386) CI check failure looks like a timeout issue unrelated to changes in this PR:
fatal: unable to access 'https://github.com/trailofbits/ruzzy.git/': Failed to connect to github.com port 443: Connection timed out
The x86_64 version of that passed. I expect running i386 again will too.
/gcbrun trial_build.py python --sanitizer coverage address --fuzzing-engine libfuzzer
@oliverchang @jonathanmetzman Only 3 projects failed the latest trial build, 2 of which are only coverage failures. pycrypto is the only fuzzer build failure (which would be a regression).
So unless I'm mistaken, all 74 failing fuzzer builds for Python projects will be fixed by this PR, and 1 new fuzzer build failure introduced.
Next Steps?
Considering the above, would you be comfortable with pinning pycrypto to the 3.8 image for now and merging this PR?
Trial Build result
Trial build end time: 2024-11-20 10:19:26.353697
https://github.com/google/oss-fuzz/pull/12027/checks?check_run_id=33239275606
| Project | Logs | Failure Type | Note |
|---|---|---|---|
mrab-regex |
Link | Coverage | Fuzzer build fails in master now. Coverage build failure on this PR is a regression. |
pycrypto |
Link | Fuzzer | This is a regression introduced by this PR. Fix might be a simple import statement update though? |
ijson |
Link | Coverage | Fuzzer build is unaffected and working fine. Coverage build failure on this PR is a regression. |