apache
Add support for Python 3.13
^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.
Looks like the only limitation now is google libraries?
No, not only - we have a number of libraries removed by "hard-coded" removals. This one needs closer look
Ok. We are close to get the PR green - with Python 3.13 support coming with:
- fab (yes - it looks like finally we can have Airflow without fab provider)
- kafka
- ydb
- yandexcloud
- apache.bem
I've been iterating over it during the last few days fixing smaller and bigger changes and while I think there are still some small things to fail, it should be "almost" green. I will add soon more detailed summary of the changes needed (there were many more changes needed than when we added Python 3.12) but if you feel like it - feel free to start reviewing it. I split this "huge" PR into separate commmits - and while we should be merging it all together, you can review commit-by-commit, it should be way easier to review as some of the changes are more mechanical and some more "real".
Getting almost there - there should be just a few tests failing left. If anyone cares to start review - that would be fantastic. I might want to extract some of these to separate PRs, as they are not necessarily tied to Python 3.13 - also I have a few TODOs left and "hacks" so maybe together we can find better solutions.
Again - the PR is split into commits where I groupped together related changes, so reviewing commit-by-commit should be rather easy.
Note - Python 3.13 was a WAY more difficult migration than Python 3.12 - there were a number of things removed/changed and small behvioral changes that we relied on, also the dependencies are catching up way later, and there are some tricky dependencies of ours that make things more complex when it comes to selecting "min" versions of those. But we are finally getting there.
I might want to extract some of these to separate PRs
I'd love some smaller "enablement" PRs to bring the review volume of this down. But I'll start reviewing it now...
Oh, boy this was a large one. Sorry for the many comments but I believe it boils down to two or thee things. Otherwise looks like a great work!
Very cool. Thank you ! I prefer too many comments than none :) Will help me to iterate on it farther for sure, and I will also attempt to extract some of the changes while doing it.
Big thanks for the first pass!
I did my first pass, nothing major, same as @jscheffl comments :)
Looking great thanks @potiuk for doing this :)
Ok. As planned and expected - the release of providers (even RC) fixed pypi constraint generation. The only remaining issue is to get https://github.com/apache/airflow/pull/52581 completed, that should remove tha last roadblock for Python 3.13 support. Thanks @eladkal ! -> @jason810496 ... happy to do more reviews and help with the "serve_logs" change :D
When running S3 tests I see
{"category": "DeprecationWarning", "message": "pathlib.Path.__enter__() is deprecated and scheduled for removal in Python 3.13; Path objects as a context manager is a no-op", "filename": "providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py", "lineno": 1568, "when": "runtest", "node_id": "providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_download_file", "param_id": null, "group": "providers", "count": 1}
{"category": "DeprecationWarning", "message": "pathlib.Path.__enter__() is deprecated and scheduled for removal in Python 3.13; Path objects as a context manager is a no-op", "filename": "providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py", "lineno": 1568, "when": "runtest", "node_id": "providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_download_file_exposes_lineage", "param_id": null, "group": "providers", "count": 1}
So i think we may have some incompatibilities that we are not covering with tests as I would have expected test failure on 3.13 - I guess we can handle it seperatly
When running S3 tests I see
{"category": "DeprecationWarning", "message": "pathlib.Path.__enter__() is deprecated and scheduled for removal in Python 3.13; Path objects as a context manager is a no-op", "filename": "providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py", "lineno": 1568, "when": "runtest", "node_id": "providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_download_file", "param_id": null, "group": "providers", "count": 1} {"category": "DeprecationWarning", "message": "pathlib.Path.__enter__() is deprecated and scheduled for removal in Python 3.13; Path objects as a context manager is a no-op", "filename": "providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py", "lineno": 1568, "when": "runtest", "node_id": "providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_download_file_exposes_lineage", "param_id": null, "group": "providers", "count": 1}So i think we may have some incompatibilities that we are not covering with tests as I would have expected test failure on 3.13 - I guess we can handle it seperatly
~Those line numbers do not match the line numbers from main so it's hard to say which tests and what is causing it. Are you sure you are running the latest main ?~
"providers/amazon/src/airflow/providers/amazon/aws/hooks/s3.py", "lineno": 1568, "when": "runtest", "node_id": "providers/amazon/tests/unit/amazon/aws/hooks/test_s3.py::TestAwsS3Hook::test_download_file_exposes_lineage", "param_id": null, "group": "providers", "count": 1}
~That does not match the method that we see in main:~
So i think we may have some incompatibilities that we are not covering with tests as I would have expected test failure on 3.13 - I guess we can handle it seperatly
I already corrected it long time ago in my PR - it did fail on 3.13:
