srinathk10

Hi @colin-ho, With this [PR](https://github.com/ray-project/ray/pull/54831/files#diff-8d080ad52f83ccc0b591e9f62c481bc6d03a5b67dcc048d5f302f89e92f13f5dL590), we are defaulting to `cloudpickle` and then fallback to legacy `json`. So you can use the corresponding `__arrow_ext_serialize__` and `__arrow_ext_deserialize__` APIs.

### Batch Inference Checkpointing - OSS design Here is the [design document](https://docs.google.com/document/d/12YNeRfZcTe-KyPo_HB71Lx5pZHJl6Evz_Idxcu4WuKk/edit?tab=t.0#heading=h.juydw89gv7zx). **Context** For long running Data pipelines deployed for Batch inferencing, each time a non-recoverable failure occurs, the pipeline...

@neuyilan Thanks for checking. We can target getting a PR out next week.