Review the Supported Python Version Policy of kedro-datasets

[noklam]

          @astrojuanlu @merelcht Let's keep this a separate discussion as there are different opinions here. I agree with @deepyaman that there is no rush to drop support when it's perfectly compatible.

Originally posted by @noklam in https://github.com/kedro-org/kedro-plugins/issues/689#issuecomment-2126820944

Quote some original discussion here:

@merelcht I think we should wait a sec with dropping 3.9 support, because I think Viz relies on kedro-datasets @rashidakanchwala. I'd like us to revisit this "imbalance" in support of Python versions across plugins and Kedro, because it has led to complications when we did that transcoding change.

@rashidakanchwala The last time this happened, we made significant code changes on Kedro-viz/Kedro-datasets in order to decouple it. And because kedro-datasets dropped support for python 3.8 whilst kedro-viz still supported it; we had to drop support for 3.8 because there were breaking changes and it didn't work for 3.8; the other alternative was to keep the old code bits to support python 3.8 users.

I think moving forward, we wouldn't be impacted as much -- only the new features in kedro-datasets wouldn't be available to python 3.9 users.

@deepyaman This shouldn't cause any issues, because pip won't let you install a library that isn't supported on the earlier version, and will simply fall back to an older version that does support the Python version.

That said, there's also no real point in dropping support without making code changes that are 3.9-incompatible.

@astrojuanlu Just to reduce the risk,

we could make a kedro-viz release before dropping Python 3.9 support here.

NEP29

In the past we adopted the NEP29 policy for kedro-datasets, mainly because of the dependencies hell makes it hard to support older Python version because we have no control of third-party libraries.

According to the policy, we should drop 3.9 support for kedro-datasets today.

Pro:

• We should drop it as the policy states
• One less version to support, strictly less burden to maintain.

Con:

• Is it necessary to drop 3.9 support when it's compatible and not causing any issue?
• How can we be sure that the transcoding compatibility issue will not happen again?

[deepyaman]

Pro:

• We should drop it as the policy states
• One less version to support, strictly less burden to maintain.

Con:

• Is it necessary to drop 3.9 support when it's compatible and not causing any issue?

Following a Python version support policy (like NEP-29) helps us in a few situations:

• If we get issues on a 3.9 build, we can not investigate, and just shut it off
• If somebody wants to use features or libraries that aren't supported on 3.9, they can do so without concern
• Any 3.9 compatibility code can be removed

It's less important to explicitly prevent people from installing on 3.9, unless you introduce features that won't work on 3.9. I usually use pyupgrade and bump the target Python version to make use of newer features, at which point you should prevent installation.

[astrojuanlu]

I'm of the opinion that the current policy is fine, and in fact I was about to open an issue about dropping 3.9.

[merelcht]

As clarified in https://github.com/kedro-org/kedro-viz/issues/2086#issuecomment-2343330747, kedro-datasets can continue with the NEP-29 policy and shouldn't affect kedro-viz. So we can close this issue and proceed with #818.