Kun-Lung Wu

icon indicating an email [email protected]

icon company IBM icon location Yorktown Heights, NY icon location A researcher at IBM Research, interested in general parallel and distributed computing, data-intensive systems and analytics.

Results 3 issues of Kun-Lung Wu

Can I use spark.createDataFrame() with a list of ObjectRef from various remote Ray workers?

18 comment

I am trying to see if I can create a Spark DataFrame with a list of ObjectRef from several remote Ray tasks that have put large objects into their local...

Support better integration between Ray and Spark in passing ObjectRef without actually moving data

3 comment

## Overview As a Codeflare user, I want to use Ray and Spark alternately to execute my end-to-end ML jobs. Some steps might be executed more efficiently using Ray, while...

ray-related

[Logging Feature] Logging INFO about completed x files in y mins should add (xx1 successfully and xx2 failed)

### Search before asking - [X] I searched the [issues](https://github.com/IBM/data-prep-lab/issues) and found no similar issues. ### Component Library/core ### Feature When running a transform, the logging INFO on my terminal...

enhancement