js-libp2p icon indicating copy to clipboard operation
js-libp2p copied to clipboard

Published 20 hours ago • verified publisher icon libp2p

Memory leak issue

Open twoeths opened this issue 1 year ago • 9 comments
trafficstars

  • Version: v1.1.1

  • Platform: Linux 5.15.0-79-generic #86-Ubuntu SMP Mon Jul 10 16:07:21 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

  • Subsystem:

Severity:

Medium

Description:

Refer to the profile taken by a lodestar beacon-node here

There are > 15k of objects retained belonging to these classes:

  • Socket
  • ConnectionImpl
  • Timeout
  • XXHandShake
  • MPlexStreamMuxer

Steps to reproduce the error:

Run a lodestar mainnet node and leave it for a long time, then monitor heap size of network thread

twoeths avatar Apr 11 '24 06:04 twoeths

I found all Sockets have a timeout and they're all destroyed Screenshot 2024-04-11 at 14 39 33

all connection statuses are also "closed" Screenshot 2024-04-11 at 14 40 39

also there are socket events attached during the MultiaddrConnection.close() function, especially the "drain" one

Screenshot 2024-04-11 at 14 42 36

it seems to me the MultiaddrConnection.close() was called, but somehow it still keep some resources especially the socket listeners

twoeths avatar Apr 11 '24 07:04 twoeths

@tuyennhv slightly off-topic but is there any chance you could share the grafana dashboard json you use over at lodestar so we can use it in ipfs/helia-http-gateway ?

SgtPooki avatar Apr 11 '24 10:04 SgtPooki

@SgtPooki here's the libp2p dashboard in lodestar https://github.com/ChainSafe/lodestar/blob/unstable/dashboards/lodestar_libp2p.json

Screenshot 2024-04-11 at 19 54 28 Screenshot 2024-04-11 at 19 54 46

twoeths avatar Apr 11 '24 12:04 twoeths

@achingbrain need to reopen until it's confirmed this is fixed in lodestar, waiting for a new release of transport-tcp to test

twoeths avatar Apr 15 '24 11:04 twoeths

@tuyennhv did you manage to test the new release?

achingbrain avatar Apr 23 '24 08:04 achingbrain

@achingbrain we tested v9.0.22 and found it does not resolve the memory leak issue. It also introduced some performance impact that caused node to forward less gossip messages to mesh peers, I'm not sure if it links to libp2p-tcp specifically or other libs during the upgrade to have to rollback in our release

see https://github.com/ChainSafe/lodestar/pull/6701/files

twoeths avatar Apr 25 '24 03:04 twoeths

I think we need to upgrade all libp2p dependencies to the latest versions and evaluate again cc @wemeetagain need to take a new profile with leaked memory for libp2p-tcp >= v9.0.22 to move this forward

twoeths avatar Apr 25 '24 03:04 twoeths

I think we need to upgrade all libp2p dependencies to the latest versions

Please do, there was a small leak fixed by @libp2p/[email protected] so that's a good idea in general.

achingbrain avatar Apr 25 '24 06:04 achingbrain

in progress PR by @wemeetagain : https://github.com/ChainSafe/lodestar/pull/6711

twoeths avatar Apr 26 '24 01:04 twoeths