besu
besu copied to clipboard
Number of peers in logs higher than maxPeers
Old title: 21.7.2 node return 5 peers on 4 node IBFT2.0 network
Description
Here is log from a node (non boot) which shows 5 peers on 4 node IBFT2.0 network. The system was upgraded from 21.1 to 21.7.2 a couple of days ago. All 4 nodes are running on a single physical server with 8GB RAM:
2021-08-29 04:49:56.053+08:00 | pool-8-thread-1 | INFO | IbftBesuControllerBuilder | Imported #93,799 / 0 tx / 0 pending / 0 (0.0%) gas / (0x9306201950f95779416cc6f8c59887aca42526fe51a8db09654e5ce3a9d36ecf)
2021-08-29 04:49:58.046+08:00 | EthScheduler-Workers-2 | INFO | PersistBlockTask | Imported #93,800 / 0 tx / 0 om / 0 (0.0%) gas / (0x1872ffef8dc9ae6bf16895f095060e1a88fd565680de743872750dffd68dd99f) in 0.000s. Peers: 5
2021-08-29 04:50:00.047+08:00 | EthScheduler-Workers-3 | INFO | PersistBlockTask | Imported #93,801 / 0 tx / 0 om / 0 (0.0%) gas / (0x2458a620019d9062e47f9c04f7682f008f98fc69ade9f03ef77263ffeca5a55a) in 0.000s. Peers: 5
2021-08-29 04:50:02.056+08:00 | EthScheduler-Workers-2 | INFO | PersistBlockTask | Imported #93,802 / 0 tx / 0 om / 0 (0.0%) gas / (0x249b2c4aa49ba5ec22d0246a4dbf2e217b45cc3ebca6e178e7d8ba2d20ba4c94) in 0.000s. Peers: 5
2021-08-29 04:50:02.058+08:00 | pool-8-thread-1 | INFO | IbftBesuControllerBuilder | Imported #93,802 / 0 tx / 0 pending / 0 (0.0%) gas / (0x249b2c4aa49ba5ec22d0246a4dbf2e217b45cc3ebca6e178e7d8ba2d20ba4c94)
2021-08-29 04:50:04.059+08:00 | pool-8-thread-1 | INFO | IbftBesuControllerBuilder | Imported #93,803 / 0 tx / 0 pending / 0 (0.0%) gas / (0x5af02576113618c80d68cc09097daaf526f73f6889789372486abf1be5250c76)
Here is debug log for boot node showing 3 peers:
2021-08-29 04:55:08.015+08:00 | pool-8-thread-1 | DEBUG | IbftRound | Received a proposal message. round=ConsensusRoundIdentifier{Sequence=93955, Round=0}
2021-08-29 04:55:08.022+08:00 | pool-8-thread-1 | DEBUG | IbftRound | Sending prepare message. round=ConsensusRoundIdentifier{Sequence=93955, Round=0}
2021-08-29 04:55:08.039+08:00 | pool-8-thread-1 | DEBUG | IbftRound | Received a prepare message. round=ConsensusRoundIdentifier{Sequence=93955, Round=0}
2021-08-29 04:55:08.039+08:00 | pool-8-thread-1 | DEBUG | IbftRound | Sending commit message. round=ConsensusRoundIdentifier{Sequence=93955, Round=0}
2021-08-29 04:55:08.045+08:00 | EthScheduler-Workers-0 | DEBUG | TransactionLogBloomCacher | Caching logs bloom for block 0x16f03.
2021-08-29 04:55:08.045+08:00 | EthScheduler-Workers-0 | INFO | PersistBlockTask | Imported #93,955 / 0 tx / 0 om / 0 (0.0%) gas / (0x986ca2cff51f96cd539ab44590057875cdf334d052cca5749d6108b18eef3cd5) in 0.000s. Peers: 3
2021-08-29 04:55:08.048+08:00 | pool-8-thread-1 | DEBUG | BaseBftController | Discarding a message which targets a height not above current chain height.
2021-08-29 04:55:08.049+08:00 | pool-8-thread-1 | DEBUG | BaseBftController | Discarding a message which targets a height not above current chain height.
2021-08-29 04:55:08.049+08:00 | pool-8-thread-1 | DEBUG | BaseBftController | Discarding a message which targets a height not above current chain height.
2021-08-29 04:55:08.049+08:00 | pool-8-thread-1 | DEBUG | BaseBftController | Discarding a message which targets a height not above current chain height.
2021-08-29 04:55:08.049+08:00 | pool-8-thread-1 | DEBUG | BaseBftController | New chain head detected (block number=93955), currently mining on top of 93954.
Here is a log from regular node showing 3 peers:
2021-08-29 04:56:24.042+08:00 | EthScheduler-Workers-3 | INFO | PersistBlockTask | Imported #93,993 / 0 tx / 0 om / 0 (0.0%) gas / (0x7bf31d4c8c023f8e190d20bd54baf1d6a2c623e479c5de6c9950ad1affb68ab8) in 0.000s. Peers: 3
2021-08-29 04:56:24.042+08:00 | pool-8-thread-1 | INFO | IbftBesuControllerBuilder | Imported #93,993 / 0 tx / 0 pending / 0 (0.0%) gas / (0x7bf31d4c8c023f8e190d20bd54baf1d6a2c623e479c5de6c9950ad1affb68ab8)
2021-08-29 04:56:26.050+08:00 | EthScheduler-Workers-1 | INFO | PersistBlockTask | Imported #93,994 / 0 tx / 0 om / 0 (0.0%) gas / (0x3fd2ce48d647aa5bc9efc4215f1fac6bd34be4a5431a1e1b55a24694c9b045d5) in 0.000s. Peers: 3
2021-08-29 04:56:26.052+08:00 | pool-8-thread-1 | INFO | IbftBesuControllerBuilder | Imported #93,994 / 0 tx / 0 pending / 0 (0.0%) gas / (0x3fd2ce48d647aa5bc9efc4215f1fac6bd34be4a5431a1e1b55a24694c9b045d5)
2021-08-29 04:56:28.047+08:00 | pool-8-thread-1 | INFO | IbftBesuControllerBuilder | Produced #93,995 / 0 tx / 0 pending / 0 (0.0%) gas / (0x221f36bdb08ec1734109a44ec0c5c4f71f46f8ffbc949c3a3f9f2b35d2837a3b)
2021-08-29 04:56:30.056+08:00 | EthScheduler-Workers-3 | INFO | PersistBlockTask | Imported #93,996 / 0 tx / 0 om / 0 (0.0%) gas / (0x16830ba3c14d2fc3d57968b86de54d51ac69a29a03e54ec8e887dd340914f657) in 0.000s. Peers: 3
2021-08-29 04:56:32.052+08:00 | EthScheduler-Workers-1 | INFO | PersistBlockTask | Imported #93,997 / 0 tx / 0 om / 0 (0.0%) gas / (0x79c1f02505dae9e7d4af21eb1934c64cbbbf8f8a103cae6b78e3e0f80c7919b2) in 0.000s. Peers: 3
Acceptance Criteria
- [Criteria 1]
Steps to Reproduce (Bug)
- [Step 1]
- [Step 2]
- [Step ...]
Expected behavior: [What you expect to happen] 3 peers on 4 node network
Actual behavior: [What actually happens] one node has 5 peers while other 3 have 3 peers.
Frequency: [What percentage of the time does it occur?]
Versions (Add all that apply)
- Software version: [
besu --version
] 21.7.2 - Java version: [
java -version
] openjdk 14 - OS Name & Version: [
cat /etc/*release
] ubuntu 20.04 - Kernel Version: [
uname -a
] - Virtual Machine software & version: [
vmware -v
] - Docker Version: [
docker version
] - Cloud VM, type, size: [Amazon Web Services I3-large]
Additional Information
Encountered the same problem on Besu 21.10. In a 4-nodes network, one node has 3 peers, one has 6 peers and two have 4 peers.
This log line comes from here https://github.com/hyperledger/besu/blob/5e0301a749ac49a611a26c1df84eee5c0ee198f6/ethereum/eth/src/main/java/org/hyperledger/besu/ethereum/eth/sync/tasks/PersistBlockTask.java#L226
Not sure how/why it would exceed the number of peers available. We can aim to add some more testing around this method.
We have observed the same behaviour. When admin_peers details are examined, the discrepancy is that we have two peers that are identical, except that one has ?disc_port=0
@RP27 Have a look at the usages of this disconnect reason code in DisconnectMessage.java ALREADY_CONNECTED((byte) 0x05), I would expect that in the above case we would get "already connected" but we aren't.
Just an update on this issue, I have been looking into the RlpxAgent code as there is a handleIncomingConnection method that is supposed to disconnect duplicate nodes from the network. The logic within the method seems to work as expected and further tests have shown that. I believe a map is being used to store the values of the nodes, which should prevent identical keys. So next, I will look into why this is occurring.
Repo with minimal reproduce instructions https://github.com/macfarla/besu-duplicate-peers
and I just ruled out IBFT2. I get the same thing with ethash.
It looks from the logs like both nodes are detecting the duplicate connection and initiating disconnect. maybe there's a race condition there.
Hi @emclab, we have had a look at that problem and we are planning to fix that issue as part of some work in the p2p area in the near future.
Just checking on the status of this work - Is https://github.com/hyperledger/besu/pull/3945 the latest approach?
@pinges - Have you found the source of the issue? I'm seeing a similar problem on some nodes on the Palm network. I ran a node locally for a bit with trace logging on and was seeing a large number of peers kept in EthPeers.connections
that were marked as "disconnected":
2022-06-09 20:30:14.661-04:00 | nioEventLoopGroup-3-9 | TRACE | EthProtocolManager | 12 EthPeers {
Peer 0x0dc8743929014993a9... PeerReputation 100, validated? true, disconnected? true,
Peer 0x0e759653a9dac12068... PeerReputation 100, validated? true, disconnected? true,
Peer 0x14601ba2d78d81d8fa... PeerReputation 100, validated? true, disconnected? true,
Peer 0x26e3ca259bce7933cc... PeerReputation 100, validated? true, disconnected? true,
Peer 0x421f71f6b88ae98083... PeerReputation 100, validated? true, disconnected? true,
Peer 0x4acab12d3e011a6ffd... PeerReputation 100, validated? true, disconnected? true,
Peer 0x4ca85a907acac09c45... PeerReputation 100, validated? true, disconnected? true,
Peer 0xbf71a1fde11fbc2104... PeerReputation 100, validated? true, disconnected? true,
Peer 0xc75062191d57de365a... PeerReputation 100, validated? true, disconnected? false,
Peer 0xcc2d5b07c0a6dc52c2... PeerReputation 100, validated? true, disconnected? true,
Peer 0xddbe110a7a2b46bbca... PeerReputation 100, validated? true, disconnected? true,
Peer 0x80af873bfc8f6ed78d... PeerReputation 100, validated? true, disconnected? false}
During normal operations, I saw this kind of behavior occassionally, but after opening / closing my laptop and moving around I ended up with that list full of stale peers ^.
One guess is that PeerConnection.equals()
is not behaving deterministically, so peers are not always being removed from EthPeers.connections on disconnect. I did notice that EnodeURLImpl.getIp() is doing a live calculation which seems like it could mess up that equality check, but I'm not sure that would explain the discrepancy for peers whose hostnames are just plain ip addresses.
@mbaxter This should get fixed with the changes that I'm doing. The main map holding the peers in EthPeers will use the peerId instead of the connection as the key. There will be another map containing potential new peers that haven't exchanged the status messages yet, and another one that will hold some peers that were already usable, but are replaced because of another connection from that peer, but they are both only holding their entries for a few seconds. These two other maps are only used when there are connection attempts from both peers at the same time, so that should not really happen in a "normal" network (just happens all the time with our acceptance tests).
@pinges - have you found the root cause of the mismatch between EthPeers.connections
and RlpxAgent.connectionsById
? RlpxAgent
is the source of truth wrt connection management, so this should just be an issue related to how connection / disconnection events are communicated and handled between these 2 objects.
Edit: Just to be clear - I'm seeing net_peerCount
(which pulls from RlpxAgent
) returning correct values while the count from EthPeers
is out of sync.
Next steps
- clean up logging on #4001
Hey @emclab we are still working on fixing the root cause here, but we think some other improvements that have been made elsewhere in peering may have reduced the incidence of this "duplicate peer" issue, to such a low level that you may not be affected by it. Are you able to try with the most recent Besu? We have just released 22.7.0-RC2
Reported by a user on 22.7.0 on mainnet https://discord.com/channels/905194001349627914/1006351975467728916
After a few days running on goerli besu 22.7.0
[TRACE] 2022-08-15 00:32:32.646 [nioEventLoopGroup-3-9] EthProtocolManager - 20 EthPeers {
PeerId 0x14601ba2d78d81d8fac9ee735cc3a4c1e49939f987adc21a6e0a391ec688a22601cc7030e5a2e9ad64ebc5e1aeb8b9dec1ccedefe041ced9614c1af6d38ee0b6, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v21.1.4/linux-x86_64/openjdk-java-11, connection 512764178, enode enode://14601ba2d78d81d8fac9ee735cc3a4c1e49939f987adc21a6e0a391ec688a22601cc7030e5a2e9ad64ebc5e1aeb8b9dec1ccedefe041ced9614c1af6d38ee0b6@34.193.87.113:30303,
PeerId 0x24de87f10d52409c348accadbfb7c7982a5bde6122e969e89bdf2323a49cd17c951f2d4cbfb7c9a3dc22d099978338ea13c47903ff8be6fa846132f1fcd0e5b9, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.1-dev-7319dbb4/linux-x86_64/openjdk-java-11, connection 1623586751, enode enode://24de87f10d52409c348accadbfb7c7982a5bde6122e969e89bdf2323a49cd17c951f2d4cbfb7c9a3dc22d099978338ea13c47903ff8be6fa846132f1fcd0e5b9@52.15.106.122:30303,
PeerId 0x2740325245eb5d34b1f6fe4758e6636969425fe1e593d7c8b46a1eeaf227b9bc91d82172bfb73ed8516fe96bfd9b4d33db61258c106d4c79e84ee28685ac29af, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.1-dev-e4b621b3/linux-x86_64/openjdk-java-11, connection 734223677, enode enode://2740325245eb5d34b1f6fe4758e6636969425fe1e593d7c8b46a1eeaf227b9bc91d82172bfb73ed8516fe96bfd9b4d33db61258c106d4c79e84ee28685ac29af@18.221.222.122:30303,
PeerId 0x284b09eb902f249e85c12a330ff88b89be3f6eacc9877e1c356152e9962910e72f746d5bfe9ae239ed93e6478deda79c8e6ffef208ce9f4f5744acabf7a14743, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.4.0-RC1/linux-x86_64/openjdk-java-11, connection 382894318, enode enode://284b09eb902f249e85c12a330ff88b89be3f6eacc9877e1c356152e9962910e72f746d5bfe9ae239ed93e6478deda79c8e6ffef208ce9f4f5744acabf7a14743@35.243.229.29:30303,
PeerId 0x2eaf83ffc16446f795a54124887388d014399deaa8110adeb0c89b1adf9783c48686caefd61f4a8455b648a84ecb9db952ea811bf14a5c68401b0d83ace07155, reputation PeerReputation 100, validated? true, disconnected? true, client: bor/v0.2.16-stable/linux-amd64/go1.18.3, connection 188072587, enode enode://2eaf83ffc16446f795a54124887388d014399deaa8110adeb0c89b1adf9783c48686caefd61f4a8455b648a84ecb9db952ea811bf14a5c68401b0d83ace07155@168.119.174.179:30000,
PeerId 0x3d4d5b69f6de9cf5c2073b56ca6e7998f910e41e5f6a8577cf17397054d1969d1c860a5f904bf2ecf260b97bcb9249c94edf4ab82c8a8fbcf375baa12f8171b2, reputation PeerReputation 100, validated? true, disconnected? false, client: Geth/v1.1.11-cf1245fe-20220629/linux-amd64/go1.16.5, connection 25860011, enode enode://3d4d5b69f6de9cf5c2073b56ca6e7998f910e41e5f6a8577cf17397054d1969d1c860a5f904bf2ecf260b97bcb9249c94edf4ab82c8a8fbcf375baa12f8171b2@3.250.56.108:0,
PeerId 0x4f3d02b456c3fd9f8c7a53f295db9fdcffbf7757ec681ddc44bfc8f38222fef81cbd7dfd4135cb11c80f7f930d047a7a9305e44d4bb62bc4cd2272fe74e83bff, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.4.0-RC2/linux-x86_64/openjdk-java-11, connection 842890299, enode enode://4f3d02b456c3fd9f8c7a53f295db9fdcffbf7757ec681ddc44bfc8f38222fef81cbd7dfd4135cb11c80f7f930d047a7a9305e44d4bb62bc4cd2272fe74e83bff@52.45.150.240:30303,
PeerId 0x523c8c89e25d6159dac2c6a41ac5fa98f7db2df44cdae1a4ae6fb739dd147ea382166e20617ead4f1c2577acb272682d9b8fd61bce8782e46554d397e0c93c2b, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.4.0/linux-x86_64/openjdk-java-11, connection 272722608, enode enode://523c8c89e25d6159dac2c6a41ac5fa98f7db2df44cdae1a4ae6fb739dd147ea382166e20617ead4f1c2577acb272682d9b8fd61bce8782e46554d397e0c93c2b@52.72.56.13:30303,
PeerId 0x5c694d2bf2b894b596fc46630066eef2d68fbc056fdde544ed0b5c4a1ecc0a81c900645b81c436c8f5c24d0e0c67a31a6aa0084e3f419b674f103f1f5eb4c3f6, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v21.10.0/linux-x86_64/openjdk-java-11, connection 1852502849, enode enode://5c694d2bf2b894b596fc46630066eef2d68fbc056fdde544ed0b5c4a1ecc0a81c900645b81c436c8f5c24d0e0c67a31a6aa0084e3f419b674f103f1f5eb4c3f6@3.109.171.197:30304,
PeerId 0x5dddbca60a3a5abc1fa66006c4e9cd44e4d60b59de9c2e38d4abab1817dbe6558abe84cd0265eed4fe1687b84b61a8df2afca01f6f02012e15e1a9fea57c6f09, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.0-dev-326113d2/linux-x86_64/openjdk-java-11, connection 1147394275, enode enode://5dddbca60a3a5abc1fa66006c4e9cd44e4d60b59de9c2e38d4abab1817dbe6558abe84cd0265eed4fe1687b84b61a8df2afca01f6f02012e15e1a9fea57c6f09@3.133.81.125:30303,
PeerId 0x5f694f6837c3983d5c29ea58fce50c3c7296ea67ddc53a52f4e00bba01473d598d450e32c76941dfe344d3d3d28af2745b9a165b26784d7696ce74835fd6e3ed, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.4.3-dev-e1d0a3d6/linux-x86_64/openjdk-java-11, connection 160178864, enode enode://5f694f6837c3983d5c29ea58fce50c3c7296ea67ddc53a52f4e00bba01473d598d450e32c76941dfe344d3d3d28af2745b9a165b26784d7696ce74835fd6e3ed@13.209.89.196:30303,
PeerId 0x5f694f6837c3983d5c29ea58fce50c3c7296ea67ddc53a52f4e00bba01473d598d450e32c76941dfe344d3d3d28af2745b9a165b26784d7696ce74835fd6e3ed, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.4.3-dev-e1d0a3d6/linux-x86_64/openjdk-java-11, connection 973513062, enode enode://5f694f6837c3983d5c29ea58fce50c3c7296ea67ddc53a52f4e00bba01473d598d450e32c76941dfe344d3d3d28af2745b9a165b26784d7696ce74835fd6e3ed@13.209.89.196:30303,
PeerId 0x6feb25e94406ad665a1b20e3cde8c2d89246558bc5614d9b0b18e7f660dee5040741c032674d0589ddd7562a2f4d635799bb8a4f2efa7c94c1735a396c6d4d94, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.0/linux-x86_64/openjdk-java-11, connection 1127470408, enode enode://6feb25e94406ad665a1b20e3cde8c2d89246558bc5614d9b0b18e7f660dee5040741c032674d0589ddd7562a2f4d635799bb8a4f2efa7c94c1735a396c6d4d94@66.115.189.188:54916,
PeerId 0x7c89c5d47311babc331454d93a4fbaca62d957535bfc0fc8adb7e07c23fd12c3cb70ad0b5cc2b5fb77353fbc1f65a1a5c3ab8b9ee6466b11860dbd40f9bb8396, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v21.1.4/linux-x86_64/openjdk-java-11, connection 256537157, enode enode://7c89c5d47311babc331454d93a4fbaca62d957535bfc0fc8adb7e07c23fd12c3cb70ad0b5cc2b5fb77353fbc1f65a1a5c3ab8b9ee6466b11860dbd40f9bb8396@34.195.196.237:30305,
PeerId 0x8deeec45c0264ac711826bbefac134958a4e5d9fee9419bd8ee27426f12d0ca7a52ad5b232234c1e1d851ae9b5211d900539ea4f10f21f7a8e2e7c807d8912ea, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.1-dev-e4b621b3/linux-x86_64/openjdk-java-11, connection 597060785, enode enode://8deeec45c0264ac711826bbefac134958a4e5d9fee9419bd8ee27426f12d0ca7a52ad5b232234c1e1d851ae9b5211d900539ea4f10f21f7a8e2e7c807d8912ea@3.17.129.175:30303,
PeerId 0x93fb884afb2f272c93e09ca9269374896caa2ad71c8d3f8cabef9d7f79e628c09ff4e09bf4ecf6d8be09c6cb06dfb12571177180ced02e2e01ab32183c796740, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.1-dev-7319dbb4/linux-x86_64/openjdk-java-11, connection 368696996, enode enode://93fb884afb2f272c93e09ca9269374896caa2ad71c8d3f8cabef9d7f79e628c09ff4e09bf4ecf6d8be09c6cb06dfb12571177180ced02e2e01ab32183c796740@18.221.231.0:30303,
PeerId 0x9db8f073e78764ca838d2f42a2f2e04c9f6449b58f4e52264cd4a2bc76d2685316f3f0f326415c0c2d3c15ef6365b68d962b92aa2e2f6b2ca5678776cf2d37cd, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.4.0/linux-x86_64/corretto-java-11, connection 1940733683, enode enode://9db8f073e78764ca838d2f42a2f2e04c9f6449b58f4e52264cd4a2bc76d2685316f3f0f326415c0c2d3c15ef6365b68d962b92aa2e2f6b2ca5678776cf2d37cd@18.170.180.71:30303,
PeerId 0x9f18e529db1d9423921e6e05b399860b9e2a0645fd846515754c3b940d0242b45331056899809e94957726592d4f0f88a3eff0754e63ac87b724db99ce7472a2, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v1.3.1/linux-x86_64/openjdk-java-11, connection 2046892439, enode enode://9f18e529db1d9423921e6e05b399860b9e2a0645fd846515754c3b940d0242b45331056899809e94957726592d4f0f88a3eff0754e63ac87b724db99ce7472a2@159.89.84.171:30303,
PeerId 0xc627dcd14e929aeca2fe0c3a8d784dfe834692c3d990fb9ea497a7dd55a18b100ca3bab2839b5c0db8bb617fa7afcb05e54c712cb325d79c8fcc1e2b1784b7d0, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v22.7.RC1-dev-d3aaca8f/linux-x86_64/corretto-java-17, connection 797543779, enode enode://c627dcd14e929aeca2fe0c3a8d784dfe834692c3d990fb9ea497a7dd55a18b100ca3bab2839b5c0db8bb617fa7afcb05e54c712cb325d79c8fcc1e2b1784b7d0@3.136.234.138:30303,
PeerId 0xe472b75f3ca9fa83e84300ec1bd433a236e28770c424061d997b88f170b4747b087fe15546cf189d6e90690a31ebd1fc23cc3c9def4ed476efbc7448a96d23aa, reputation PeerReputation 100, validated? true, disconnected? true, client: besu/v21.10.0/linux-x86_64/openjdk-java-11, connection 844363741, enode enode://e472b75f3ca9fa83e84300ec1bd433a236e28770c424061d997b88f170b4747b087fe15546cf189d6e90690a31ebd1fc23cc3c9def4ed476efbc7448a96d23aa@3.111.65.29:30306}
#4304 should help with this issue
This issue should be resolved by #4304