datafusion-ballista
datafusion-ballista copied to clipboard
refine the scheduler state cache layer
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
After introducing #59, the previous cache layer is almost ineffective, which will downgrade the task scheduling performance very much especially for scheduling thousands of tasks.
Describe the solution you'd like
It's better to make the cache layer effective again.
Describe alternatives you've considered
Additional context