incubator-uniffle
incubator-uniffle copied to clipboard
[Improvement] Big Partition require buffer fail, and improve it.
Code of Conduct
- [X] I agree to follow this project's Code of Conduct
Search before asking
- [X] I have searched in the issues and found no similar issues.
What would you like to be improved?
When there is big partition require buffer, require buffer fails and may wait.
Until there is flush:
How should we improve?
Provide better default arguement and advise.
Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
Machine 128G memory, rss Xmx 75G, -XX:MaxDirectMemorySize=20G.
Now, rss.server.shuffleBufferManager.trigger.flush.interval is default value and flush check thread is close, and we want to set it to 180 and enable flush check.
Now the huge partition is default value 20G, and it is bigger for me, I want to adjust to 8G according to the log: (Make it flush early)
Above is First Appear.