raydp icon indicating copy to clipboard operation
raydp copied to clipboard

Refactor object holder to be able to remove reference

Open kira-lin opened this issue 1 year ago • 0 comments

In order to be able to access objects after spark executors shutdown, we can assign the ownership to RayDPSparkMaster. But for now, we cannot cleanup the reference, so objects created such way cannot be free. This will be a memory issue when dealing with large amount of data.

This PR adds a function to delete such references. And to be able to do this, we use object.hex() (string repr of object id) as key. This function needs to be called manually by user when user finish using the object.

kira-lin avatar Mar 31 '23 06:03 kira-lin