Kun-Lung Wu

> The rest lgtm. I notice the serve.start is not in the code. Could you confirm that it's working? (I see your comment that it's not working). > > Btw,...

This feature can be supported via the [Ray Datasets](https://docs.ray.io/en/master/data/dataset.html) (currently on alpha with some missing methods, such as `ray.data.from_spark()` and `ds.to_spark()`). For example, to exchange data from Spark to Pandas,...

@kira-lin thanks for the comments. I wonder if you could clarify two questions. (1) Is there any connection between `MLDataset` and `ray.cloudpickle.dumps()`? I assume they are not related. (2) Your...

@kira-lin Thanks very much. Appreciated the help. Here is what I did, but still not working as I expected. ``` import cloudpickle @ray.remote def create_small_dataframe(i): pdf = pd.DataFrame(data=np.random.randint(5*i, size=(3, 4)))...

@kira-lin thanks very much. I will take a look at the [#166 ](https://github.com/oap-project/raydp/pull/166) PR. The idea is to pass various `Pandas dataframes` (created by remote functions) via `ObjectRef` to `Spark...

@kira-lin Thanks for your help in getting Spark to successfully create a `dataframe` with a list of `serialized objectref`. After that, I would like to, on the Spark side, get...

@kira-lin I think the problem is that an executor of PySpark doesn't have access to Ray, even if I do a `ray.init(address='auto', ignore_rayinit_error = True) `before `myrdd = sdf.rdd.flatMap(lambda x:...

@kira-lin Many thanks again. I tried your suggestion of initiating `ray `in the `map_func()`. I had to add a `return()` to the `map_func()`, however. Here is what I did: ```...

Here is what I tried: ``` obj_ref_schema = StructType([StructField('Pandas_df_ref', BinaryType(), True)]) sdf = spark.createDataFrame(data = obj_ref_list, schema = obj_ref_schema) sdf.show() def map_func(x): # command for executors to connect to ray...

@kira-lin thanks. I think I got it to work in my `map_func()` with the following: ``` def map_func(blocks): # command for executors to connect to ray cluster # ray.init will...