[fullnode] Synchronization problem if lack of memory

[Nelrann]

Steps to Reproduce Issue

Start a full node with docker and limit resource to 8Go

Expected Result

According to the documentation, a fullnode should sync with 8Go https://docs.sui.io/build/fullnode#hardware-requirements

Actual Result

Fullnode stops syncing when there are no more resources available

System Information

• OS: 20.04

[pops-one]

we can confirm theres a memory leak or a race condition here, our machine has 32gb RAM and its using all of it (98%)

[pops-one]

to update, the race condition we met is not from a new node, our node has been running for days and memory usage slowly increased

its on a dedicated 32gb ram, i7 and 1tb nvme ssd with 1gbps connection

using binaries with systemd on Ubuntu 20.04.4 LTS

[Nelrann]

CPU/Memory usage

[gdanezis]

Many thanks for your bug report. We are looking into it.

[pops-one]

Just a follow up, we restarted our systemd and as thought the RAM usage fell down significantly (6% utilization). Now after 3h or so, its crawled back to 38% utilization already, which is already over 8GB

[mystenmark]

Thanks for the report, I believe I've reproduced the issue and should have a fix soon.

[mystenmark]

There are a number of bug fixes in main right now that will be in the next release - among them will be #3489 which I think may have been the root cause of the problem you reported.

[Nelrann]

thank you @mystenmark, I'm waiting for the changes to be on the devnet branch to give you feedback

[Nelrann]

@mystenmark The latest updates seem to consume all the memory of my VPS

edit : the latest docker image sha256:40fa03646d559aa6472db50859b9e8a3afdcd2faa522da263f527d9435de04ee seems to fix the problem

[Nelrann]

@mystenmark @gdanezis I just did a test again and it seems that the memory exceeds 8G. this problem seems to happen as soon as there are more than 70K transactions to sync