Yizhong Zhang

Revert "Revert "[Core] Support Arrow zerocopy serialization in object…

3

… store (#35110)" (#36000)" This reverts commit 822904b312423010682fa92f4085c05e08819337. ## Why are these changes needed? Last PR was reverted since `test_advanced_9` failed. However, I tested `test_advanced_9` on local device and it...

Runtime env_vars cannot be passed down to executors

1

When we use `ray.init(runtime_env={"env_vars":"xxx"})`, executors cannot get "env_vars" while it is critical for executors in many scenarios. It is because `RayAppMaster` will raise a new job which is separated from...

We initialize a spark instance by `raydp.init_spark('benchmark', 1, 5, '30G', configs={"spark.default.parallelism": 50})`, Then save the dataframe to object store: ```python blocks, _ = raydp.spark.dataset._save_spark_df_to_object_store(df, False) ds = ray.data.from_arrow_refs(blocks) ``` Problem...

Together with this PR of Ray : [[Core] Support Arrow zerocopy serialization in object store #35110](https://github.com/ray-project/ray/pull/35110)