Feature request: pyspark 3.0.0 version support

[halilduygulu]

Please fill out the form below.

System Information

• Spark or PySpark:
• SDK Version:
• Spark Version:
• Algorithm (e.g. KMeans):

Describe the problem

When is pyspark 3.0.0 version support coming? Can we have an estimate to plan update in projects? Right now sagemaker-pyspark lib is not working with latest pyspark as expected.

• Exact command to reproduce:

[metrizable]

@halilduygulu Thanks for using Amazon SageMaker.

Currently, we don't have an estimate on upgrading our pyspark dependency to 3.0.0.

We are always re-evaluating our backlog of features based on customer requests, so we appreciate the feedback on this feature.

Let us know if there is anything else we can be an assistance of.

[odesiuz]

Just to second what @halilduygulu has asked, I have been waiting for sagemaker to support pyspark 3.0.0. It will be nice to update the dependency as I guess some people will be eager to try out some new feat to see the improvements in spark 3.0.0.

Thanks

[filipe-plutoflume]

Commenting here to follow the thread.

[kapilkd13]

any update on this?

[jayantshekhar]

Looking forward to support for Spark 3.x. Would be helpful to know the latest on it.

[e13h]

Hopefully PR #135 receives some attention, since it would close this issue.

[sklingel]

Any updates on this? The pyspark=2.4.0 is soon to be 3 years old.

[mattiamatrix]

Hello, any updates?

[tbugfinder]

Hello, any updates?

[dantavo]

any update on this?