sagemaker-python-sdk icon indicating copy to clipboard operation
sagemaker-python-sdk copied to clipboard

Published 20 hours ago • verified publisher icon aws

Why the supported PySpark version (2.4) is so outdated given spark 3.2 is available?

Open jtzhang17 opened this issue 3 years ago • 2 comments

Very simple request: is it possible to keep track the most recent major version of PySpark and support it in PySparkProcessor? If the PySpark is so outdated in PySparkProcessor, I doubt if there are many users willing to use this PySparkProcessor.

jtzhang17 avatar Mar 10 '22 07:03 jtzhang17

I don't think you are limited to 2.4 anymore. If you look at the supported versions here: https://github.com/aws/sagemaker-python-sdk/blob/master/src/sagemaker/image_uri_config/spark.json

It appears that v 3.1 is supported under py 3.7

jperuggia avatar Apr 12 '22 00:04 jperuggia

Thanks for creating issue. We are currently supporting version 3.1. Does that work for you ?

navinsoni avatar Jul 06 '22 18:07 navinsoni

Closing this issue as per @navinsoni's comment. Feel free to re-open if there are any further issues

akrishna1995 avatar Dec 28 '23 22:12 akrishna1995