[Task]: Support Python 3.12

[AnandInguva]

What needs to happen?

Add support for Python 3.12

Issue Priority

Priority: 2 (default / most normal work should be filed as P2)

Issue Components

• [X] Component: Python SDK
• [ ] Component: Java SDK
• [ ] Component: Go SDK
• [ ] Component: Typescript SDK
• [ ] Component: IO connector
• [ ] Component: Beam YAML
• [ ] Component: Beam examples
• [ ] Component: Beam playground
• [ ] Component: Beam katas
• [ ] Component: Website
• [ ] Component: Spark Runner
• [ ] Component: Flink Runner
• [ ] Component: Samza Runner
• [ ] Component: Twister2 Runner
• [ ] Component: Hazelcast Jet Runner
• [ ] Component: Google Cloud Dataflow Runner

[potiuk]

Hey. Friendly question here.

After Datastax fixed their libev support for cassandra-driver (testing it in https://github.com/apache/airflow/pull/38314) , Papermill and Apache Beam are two last community Airflow Providers that have no Python 3.12 support.

We are just about to release Airflow 2.9.0 where Python 3.12 support is enabled and also default airflow image is going to by Python 3.12, so Apache Beam will be uninstallable there and unusable - due to conflicting dependencies that have no Python 3.12 support (it will still work on Python 3.8 - 3.11).

I looked at the 2.55.0rc3 just uploaded - and it does not seem to come with Python 3.12 support, so I'd love to know if Python 3.12 support is planned any time soon.

[liferoad]

This will be worked on in Q2. Thanks.

[tvalentyn]

TODO:

Update all GHA tests definitions, see:

grep -rn . --include="*.yml" -e "python_version:"
grep -rn . --include="*.yml" -e "python_version: \['3.8','3.9','3.10','3.11'\]" 

./.github/workflows/beam_PreCommit_Python_Transforms.yml:67:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml:68:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PreCommit_Python_Examples.yml:67:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PostCommit_Python_Examples_Direct.yml:66:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml:66:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_Python_ValidatesContainer_Dataflow_ARM.yml:57:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PreCommit_Python.yml:67:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PreCommit_Python_Dataframes.yml:67:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PreCommit_Python_Runners.yml:67:        python_version: ['3.8','3.9','3.10','3.11']
./.github/workflows/beam_PreCommit_PythonDocker.yml:67:        python_version: ['3.8','3.9','3.10','3.11']

[potiuk]

Looking forward to enabling apache-beam provider for Airflow to 3.12 finally !