[FEATURE] Requesting brickflow support for python versions 3.11 and 3.12

[raajaaDheeRaajaa]

Is your feature request related to a problem? Please describe. Currently, the compatible versions of python for brickflow are pinned to 3.8, 3.9 & 3.10 https://github.com/Nike-Inc/brickflow/blob/09a2af92a2d9730189ea71277b59e52878caaee6/pyproject.toml#L37

Requesting to add support for python versions 3.11 & 3.12, for users using pyspark version >=3.5

Cloud Information

• [ ] AWS
• [ ] Azure
• [ ] GCP
• [X] Other

Describe the solution you'd like A clear and concise description of what you want to happen.

Describe alternatives you've considered Updating the acceptable python version from >=3.8,<3.11 to >=3.8,<=3.12 here: https://github.com/Nike-Inc/brickflow/blob/09a2af92a2d9730189ea71277b59e52878caaee6/pyproject.toml#L37

Additional context This change will allow users running newer version of python with new features and bug fixes, able to use brickflow.

[t1ffanyt]

Hello, I can take up this issue and test brickflow with the updated python versions.

[stikkireddy]

Hey team, intentially we are supporting 3.8 - 3.10 as those are the system versions supported by databricks runtimes. 14.3 LTS beta is provided with Python: 3.10.12 https://docs.databricks.com/en/release-notes/runtime/14.3lts.html#system-environment

The risk of supporting higher versions is it may encourage users to use newer 3.11, 3.12 and 3.13 features which may not work on 3.10. The user will not run into this without, actually running their workflow.

I am not opposed to this change, as the library should work in higher versions (since new versions of python contain all the previous features) but I do not know of a good way to indicate to the user to not use 3.11+ features. Let me know if you have any ideas around that.

[Mohansaik]

@stikkireddy that's true but for the Databricks runtime 16.1,16.2 and 16.3 it uses 3.12.3 python version, due to that we can't able to run Brickflow code in those clusters. due to the python version constraint not able to utilize features like string collation, etc . Please try to release new version that Brickflow supports python 3.12.3.

[nogitting]

@Mohansaik, the idea is to align with the LTS Runtimes (which is a best practice). And none of those you mentioned is one. However, 16.4 LTS has just been released. So, Brickflow shall add support for Python 3.12 in the next release (v1.4.0).

One clarification though, Python does not adopt SemVer. For instance, minor releases may include breaking changes.

[Mohansaik]

@nogitting, Thanks for your response, could you please let us know the expected release date for Brickflow v1.4.0, which includes support for Python 3.12?