kedro icon indicating copy to clipboard operation
kedro copied to clipboard

Remove `setuptools` from `spaceflights-pyspark-viz` starter on `pyspark` release

Open ankatiyar opened this issue 10 months ago • 0 comments

Description

While adding python 3.12 support to kedro-starters https://github.com/kedro-org/kedro-starters/pull/219, this issue came up. Full discussion in the PR.

  • Python 3.12 got rid of distutils https://docs.python.org/3.12/whatsnew/3.12.html#distutils
  • pyspark uses distutils in their code which they have fixed but will be released in 4.0.0
  • Temp fix - add setuptools to requirements.txt in spaceflights-pyspark-viz starter for only `python_version>=3.12" https://github.com/kedro-org/kedro-starters/pull/220

When pyspark releases their 4.0.0 version

  • Remove kedro-datasets pyspark pin which is currently <4.0.0 I believe
  • Remove setuptools from the requirements.txt

ankatiyar avatar Apr 24 '24 12:04 ankatiyar