Airflow Python 3.12 support

[dirrao]

Description

Right now, airflow is not officially supporting the Python 3.12 version

Use case/motivation

Python software stack update.

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

Code of Conduct

• [X] I agree to follow this project's Code of Conduct

[Taragolis]

This kind of task not a straight forward, required to someone check what kind of core / providers dependencies not ready for 3.12 yet, or we use outdated dependencies or some deprecated stuff:

At current moment we know at lest incompatible with that pendulum 2 doesn't support of python 3.12, but this one could be solved by https://github.com/apache/airflow/pull/36281

[potiuk]

At current moment we know at lest incompatible with that pendulum 2 doesn't support of python 3.12, but this one could be solved by https://github.com/apache/airflow/pull/36281

Yeah. I believe this is the only "serious" blocker. We have option to exclude certain providers if they turn out to be incompatible ("excluded-python-versions" flag in provider.yaml) and we've done that in the past. So I'd say @dirrao if you want to take the lead on that - open a PR with added Python 3.12 in Breeze and we can keep it open and rebased until all the tests pass (might need to exclude some providers).

I think it's about the right time to attempt it - usually it takes 3-4 months for most important dependencies to catch up. We might also need to bump some more deps, not only Pendulum, but yeah - Pendulum is the "hard" blocker.

[dirrao]

At current moment we know at lest incompatible with that pendulum 2 doesn't support of python 3.12, but this one could be solved by #36281

Yeah. I believe this is the only "serious" blocker. We have option to exclude certain providers if they turn out to be incompatible ("excluded-python-versions" flag in provider.yaml) and we've done that in the past. So I'd say @dirrao if you want to take the lead on that - open a PR with added Python 3.12 in Breeze and we can keep it open and rebased until all the tests pass (might need to exclude some providers).

I think it's about the right time to attempt it - usually it takes 3-4 months for most important dependencies to catch up. We might also need to bump some more deps, not only Pendulum, but yeah - Pendulum is the "hard" blocker.

Yes. I started looking into updating Breeze.

[dirrao]

I have updated the breeze. However when I run this test with 3.12, I am seeing python3.12 image not found. How to publish this image?

Error response from daemon: No such image: ghcr.io/apache/airflow/main/ci/python3.12:latest

[potiuk]

You should be able to build it locally with breeze ci-image build --python 3.12 (after you add it to configuration)

[potiuk]

You might see some missing cache etc. but they should be just warnings - and you can always disable cache with build command flags (use --help to see the flags)

[dirrao]

I can run after making the changes in Breeze. However, I faced two problems as follows.

1. The build is looking for the remote constraints file. I can resolve it by using a local constraints file. https://raw.githubusercontent.com/apache/airflow/constraints-3.12.txt/constraints-source-providers-3.12.txt
2. Pip is not able to resolve the remote apache airflow package version. Not sure we can if can bypass it. #0 22.01 ERROR: Package 'apache-airflow' requires a different Python: 3.12.1 not in '<3.12,~=3.8'

Can you let me know the next steps?

[potiuk]

Oh yeah.

1. I just solved the first one by pushing copy of 3.11 constraints which are "good enough" to start with - this should be fixed now
2. the second is I think the result of the build optimilsation we have for the build to use main for our. I opened PR https://github.com/apache/airflow/pull/36595 to remove the limt from main and once merged this problem should be gone.

[potiuk]

For 1: see this commit https://github.com/apache/airflow/commit/3f3ed8118aaf3630e986d98c7015eaa6073caded

(and generally main constraints history is here https://github.com/apache/airflow/commits/constraints-main/)

[potiuk]

Merged. You can continue your quest @dirrao

[dirrao]

Now, I can make progress. However, the 3.12 uploaded dependency constraint files are old as compared to 3.11. For the time being I am continuing with 3.11 as the base. As of now, I am stuck at the below error. Looks like I am facing the below error mentioned in the below stack overflow thread. AttributeError: module 'pkgutil' has no attribute 'ImpImporter'. Did you mean: 'zipimporter'? https://stackoverflow.com/questions/77364550/attributeerror-module-pkgutil-has-no-attribute-impimporter-did-you-mean

[potiuk]

Re: old dependencies: This is no problem. You should always build your image with --upgrade-to-newer-dependencies locally - this will autoamatically get your dependencies upgraded to latest matching Airflow specification.

What you also want, is to make single modification (might be an added coomment) to setup.py or pyproject.toml and open a PR with your changes (and 3.12 support added) . What this will cause is that your PR will get the image built with --upgrade-to-newer-dependencies flag automatically.

I think once you open such PR, you can easier share the problems you will experience.

[Froskekongen]

Any progress on this @dirrao?

[potiuk]

Any progress on this @dirrao?

Python 3.12 is currently being worked on in https://github.com/apache/airflow/pull/36755 . There is one thing left in order to merge it - namely converting universal_pathlib to use the new Python 3.12 ready 0.2.0 version (you can see detail and links in the PR). Would you like to help with that @Froskekongen ? It waits for someone to do it.

[potiuk]

Well. It's going to be closed when the PR adding Python 3.12 is merged :)

[potiuk]

Rebsed and runnning the PR https://github.com/apache/airflow/actions/runs/7972889219

[eladkal]

Fixed in https://github.com/apache/airflow/pull/38025