raydp icon indicating copy to clipboard operation
raydp copied to clipboard

Published 20 hours ago • verified publisher icon oap-project

Fault tolerance of dataset converted from dataframe

Open kira-lin opened this issue 2 years ago • 0 comments

This PR implements fault tolerance of ray dataset converted from spark dataframe using raydp. This requires spark session to be holded by an ray actor. This PR also introduces such a spark app template class to make users' deployment easier.

Note: this PR now use ray.get(copy) to implement. It is not ready to use in large-scale workload.

Closes #207

kira-lin avatar Apr 12 '22 05:04 kira-lin