data-prep-kit icon indicating copy to clipboard operation
data-prep-kit copied to clipboard
Published 1 week ago • verified publisher icon data-prep-kit

[Feature] Checkpointing in the spark runtime

Open daw3rd opened this issue 1 year ago • 0 comments

Search before asking

  • [X] I searched the issues and found no similar issues.

Component

Library/core

Feature

The Ray-based runtime includes checkpoint/restart for large datasets. We would like to have this same capability in the spark runtime/launcher.

Are you willing to submit a PR?

  • [ ] Yes I am willing to submit a PR!

daw3rd avatar May 28 '24 18:05 daw3rd