incubator-uniffle icon indicating copy to clipboard operation
incubator-uniffle copied to clipboard
verified publisher icon apache

[Bug] Off-heap memory grows

Open dingshun3016 opened this issue 2 years ago • 3 comments

Code of Conduct

Search before asking

  • [X] I have searched in the issues and found no similar issues.

Describe the bug

Off-heap memory growth with latest version of code rss.rpc.server.type GRPC MAX_DIRECT_MEMORY_SIZE = 70G 1705312223134

Affects Version(s)

master

Uniffle Server Log Output

No response

Uniffle Engine Log Output

No response

Uniffle Server Configurations

No response

Uniffle Engine Configurations

No response

Additional context

No response

Are you willing to submit PR?

  • [X] Yes I am willing to submit a PR!

dingshun3016 avatar Jan 15 '24 09:01 dingshun3016

When I set MAX_DIRECT_MEMORY_SIZE=30G, jvm_buffer_pool_used_bytes{pool="direct"} can reach 30G

截屏2024-01-15 18 37 38

dingshun3016 avatar Jan 15 '24 10:01 dingshun3016

Do you have any ideas about this issue? @zuston

dingshun3016 avatar Jan 15 '24 10:01 dingshun3016

Maybe you can try the latest version, it might have something to do with GRPC internally.

rickyma avatar Feb 13 '24 06:02 rickyma

You can try GRPC_NETTY. Closed due to https://github.com/apache/incubator-uniffle/issues/1651.

rickyma avatar Jun 19 '24 09:06 rickyma