incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard

[Improvement] Big Partition require buffer fail, and improve it.

Open lifeSo opened this issue 1 year ago • 3 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

What would you like to be improved?

When there is big partition require buffer, require buffer fails and may wait. image

Until there is flush: image

How should we improve?

Provide better default arguement and advise.

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

lifeSo avatar Jan 29 '24 03:01 lifeSo

Machine 128G memory, rss Xmx 75G, -XX:MaxDirectMemorySize=20G.

lifeSo avatar Jan 29 '24 03:01 lifeSo

Now, rss.server.shuffleBufferManager.trigger.flush.interval is default value and flush check thread is close, and we want to set it to 180 and enable flush check.

lifeSo avatar Jan 29 '24 03:01 lifeSo

Now the huge partition is default value 20G, and it is bigger for me, I want to adjust to 8G according to the log: (Make it flush early)

image Above is First Appear.

lifeSo avatar Jan 30 '24 03:01 lifeSo