raydp icon indicating copy to clipboard operation
raydp copied to clipboard

Published 20 hours ago • verified publisher icon oap-project

update to use Spark 3.3

Open kira-lin opened this issue 2 years ago • 3 comments

kira-lin avatar Jul 12 '22 08:07 kira-lin

better to use spark shim in 3.3

kira-lin avatar Jul 13 '22 02:07 kira-lin

We can drop Python 3.6. PySpark 3.3 requires Python 3.7+.

carsonwang avatar Jul 14 '22 06:07 carsonwang

ok, but we'll need a shim layer to add support for 3.3. I've told @KepingYan to look into it. I 'll drop python 3.6 here, and it seems like raydp can work with ray 1.13.

kira-lin avatar Jul 14 '22 06:07 kira-lin

LGTM

carsonwang avatar Oct 18 '22 08:10 carsonwang