go-ethereum icon indicating copy to clipboard operation
go-ethereum copied to clipboard

Published 20 hours ago • verified publisher icon ethereum

Optimize the memory allocation in trie operation

Open rjl493456442 opened this issue 6 months ago • 3 comments 

Build ID: 15e9e5882df65907942fa3b5d1ec150f3a7d3b90
Type: inuse_space
Time: 2025-05-11 15:03:46 CST
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) alloc_space
(pprof) top
Showing nodes accounting for 75.21TB, 54.11% of 138.99TB total
Dropped 2219 nodes (cum <= 0.69TB)
Showing top 10 nodes out of 231
      flat  flat%   sum%        cum   cum%
   25.46TB 18.32% 18.32%    29.25TB 21.05%  github.com/ethereum/go-ethereum/trie.(*hasher).hashFullNodeChildren
   12.96TB  9.32% 27.64%    25.60TB 18.42%  github.com/ethereum/go-ethereum/trie.decodeFull
   12.65TB  9.10% 36.74%    12.65TB  9.10%  github.com/ethereum/go-ethereum/trie.decodeRef
    7.37TB  5.30% 42.05%     7.37TB  5.30%  github.com/ethereum/go-ethereum/rlp.(*encBuffer).makeBytes
    6.18TB  4.45% 46.49%     6.18TB  4.45%  github.com/ethereum/go-ethereum/trie.(*tracer).onRead
    2.42TB  1.74% 48.24%     3.96TB  2.85%  github.com/ethereum/go-ethereum/core/state.newObject
    2.42TB  1.74% 49.97%     2.42TB  1.74%  golang.org/x/crypto/sha3.NewLegacyKeccak256 (inline)
    2.01TB  1.44% 51.42%     2.51TB  1.81%  github.com/ethereum/go-ethereum/core/state.(*stateObject).finalise
    1.96TB  1.41% 52.83%     4.30TB  3.10%  github.com/ethereum/go-ethereum/core/state.(*stateObject).GetCommittedState
    1.78TB  1.28% 54.11%     1.78TB  1.28%  github.com/ethereum/go-ethereum/core/state.(*journal).append

The memory profile shows that most of memory allocation is spent on several functions, such as trie node hashing. The current speed for memory allocation is about 5-600 MB/s which is insane. These part should be investigated for a memory allocation friendly change.

rjl493456442 avatar May 15 '25 05:05 rjl493456442

May I take on it?

nadtech-hub avatar May 15 '25 17:05 nadtech-hub 

Build ID: 15e9e5882df65907942fa3b5d1ec150f3a7d3b90
Type: inuse_space
Time: 2025-05-11 15:03:46 CST
Entering interactive mode (type "help" for commands, "o" for options)
(pprof) alloc_space
(pprof) top
Showing nodes accounting for 75.21TB, 54.11% of 138.99TB total
Dropped 2219 nodes (cum <= 0.69TB)
Showing top 10 nodes out of 231
      flat  flat%   sum%        cum   cum%
   25.46TB 18.32% 18.32%    29.25TB 21.05%  github.com/ethereum/go-ethereum/trie.(*hasher).hashFullNodeChildren
   12.96TB  9.32% 27.64%    25.60TB 18.42%  github.com/ethereum/go-ethereum/trie.decodeFull
   12.65TB  9.10% 36.74%    12.65TB  9.10%  github.com/ethereum/go-ethereum/trie.decodeRef
    7.37TB  5.30% 42.05%     7.37TB  5.30%  github.com/ethereum/go-ethereum/rlp.(*encBuffer).makeBytes
    6.18TB  4.45% 46.49%     6.18TB  4.45%  github.com/ethereum/go-ethereum/trie.(*tracer).onRead
    2.42TB  1.74% 48.24%     3.96TB  2.85%  github.com/ethereum/go-ethereum/core/state.newObject
    2.42TB  1.74% 49.97%     2.42TB  1.74%  golang.org/x/crypto/sha3.NewLegacyKeccak256 (inline)
    2.01TB  1.44% 51.42%     2.51TB  1.81%  github.com/ethereum/go-ethereum/core/state.(*stateObject).finalise
    1.96TB  1.41% 52.83%     4.30TB  3.10%  github.com/ethereum/go-ethereum/core/state.(*stateObject).GetCommittedState
    1.78TB  1.28% 54.11%     1.78TB  1.28%  github.com/ethereum/go-ethereum/core/state.(*journal).append

The memory profile shows that most of memory allocation is spent on several functions, such as trie node hashing. The current speed for memory allocation is about 5-600 MB/s which is insane. These part should be investigated for a memory allocation friendly change.

Eikyarkway avatar May 16 '25 16:05 Eikyarkway

https://github.com/ethereum/go-ethereum/pull/31902 for addressing hashFullNodeChildren

rjl493456442 avatar May 29 '25 08:05 rjl493456442