apache
Cleanup job/stage status from TaskManager and clean up shuffle data after a period after JobFinished
Is your feature request related to a problem or challenge? Please describe what you are trying to do. A clear and concise description of what the problem is. Ex. I'm always frustrated when [...] (This section helps Arrow developers understand the context and why for this feature, in addition to the what)
Today, there is no clean up logic to remove those job/stage status from StateBackend, the disk space might be exhausted quickly in a busy cluster.
Describe the solution you'd like A clear and concise description of what you want to happen.
Describe alternatives you've considered A clear and concise description of any alternative solutions or features you've considered.
Additional context Add any other context or screenshots about the feature request here.
