kedro
kedro copied to clipboard
Remove `setuptools` from `spaceflights-pyspark-viz` starter on `pyspark` release
Description
While adding python 3.12 support to kedro-starters
https://github.com/kedro-org/kedro-starters/pull/219, this issue came up. Full discussion in the PR.
- Python 3.12 got rid of
distutils
https://docs.python.org/3.12/whatsnew/3.12.html#distutils -
pyspark
usesdistutils
in their code which they have fixed but will be released in 4.0.0 - Temp fix - add
setuptools
torequirements.txt
inspaceflights-pyspark-viz
starter for only `python_version>=3.12" https://github.com/kedro-org/kedro-starters/pull/220
When pyspark
releases their 4.0.0 version
- Remove
kedro-datasets
pyspark
pin which is currently <4.0.0 I believe - Remove
setuptools
from therequirements.txt