Ken, Wang comments

Results 66 comments of


                                            Ken, Wang

Graceful shutdown: Handle `SIGTERM`

@thinkharderdev Sorry, I didn't get a chance to test it in a K8s cluster. I will take a look today.

Graceful shutdown: Handle `SIGTERM`

https://github.com/apache/arrow-ballista/pull/278

Clean up job data on both Scheduler and Executor

@thinkharderdev @yahoNanJing @Ted-Jiang @andygrove

Clean up job data on both Scheduler and Executor

> I am constantly filling my disk up with shuffle files so would love to see us get this merged before the 0.9.0 release. > > @mingmwang Could you rebase...

Clean up job data on both Scheduler and Executor

@andygrove @yahoNanJing @Ted-Jiang @yahoNanJing BTW, in this PR, the job data in the state store will also be deleted after 300s. I think we need a following PR to move...

Clean up job data on both Scheduler and Executor

> @mingmwang could you fix the conflicts here when you have the time so that we can merge this? Sure, I will fix the conflicts tomorrow.

Clean up job data on both Scheduler and Executor

Resolved conflicts.

Cleanup job/stage status from TaskManager and clean up shuffle data after a period after JobFinished

https://github.com/apache/arrow-ballista/issues/9

Task level retry and Stage level retry

@thinkharderdev @andygrove @yahoNanJing Please help to review my PR.

Task level retry and Stage level retry

Added 10+ UTs to cover different cases.