juicefs icon indicating copy to clipboard operation
juicefs copied to clipboard

Published 20 hours ago • verified publisher icon juicedata

S3 gateway memory skyrocketing issue

Open HavAnIdea opened this issue 11 months ago • 4 comments

Discussed in https://github.com/juicedata/juicefs/discussions/4502

Originally posted by HavAnIdea March 14, 2024 I encountered a memory skyrocketing issue when I initiated the S3 gateway for reading and writing through Spark. However, when I use DataX or JuiceFS sync, there are no issues encountered.

How should I troubleshoot the issue?

usually 1

reading and writing through Spark 2

HavAnIdea avatar Mar 15 '24 05:03 HavAnIdea

gateway log image

HavAnIdea avatar Mar 15 '24 05:03 HavAnIdea

I also met a similar memory issue when using gateway and have fixed it by #4507. Could you try it?

SandyXSD avatar Mar 15 '24 06:03 SandyXSD

I also met a similar memory issue when using gateway and have fixed it by #4507. Could you try it?

I see that your commit has been merged into the main branch. Should I compile and use it locally? I'm not sure if setting flags would be helpful for reading and writing with Spark, but I can give it a try.

HavAnIdea avatar Mar 15 '24 06:03 HavAnIdea

Can you grab the heap profile for analysis?

davies avatar Mar 21 '24 09:03 davies