ChainSafe
`Static key doesn't match to peer that signed payload` error on connection in hive docker
Describe the bug Lodestar fails to connect with bootnode peer in hive docker environment (using https://github.com/ethereum/hive/pull/577).
The second node fails to connect with the following error:
Jul-05 20:10:30.478[NETWORK] [34mdebug[39m: Error dialing discovered peer peer=16...wncgtg Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Static key doesn't match to peer that signed payload!
AggregateError: Error: Error occurred during XX handshake: Error occurred while verifying signed payload: Static key doesn't match to peer that signed payload!
at maybeSettle (/usr/app/node_modules/p-some/index.js:31:11)
at /usr/app/node_modules/p-some/index.js:69:23
at processTicksAndRejections (node:internal/process/task_queues:96:5)
ENR is collected from the boot node from the /eth/v1/node/identity endpoint, and appended as command line argument for the second node:
https://github.com/marioevz/hive/blob/f345b86a6df51ff32e66f8b0e58d24b7df7ae574/clients/lodestar-bn/lodestar_bn.sh#L37
Expected behavior Both nodes should be connected to perform the test.
Steps to Reproduce
Use https://github.com/marioevz/hive/tree/client-lodestar branch to run the eth2/engine hive simulator:
./hive --client go-ethereum,lodestar-bn,lodestar-vc --sim eth2/engine --sim.loglevel 5
Desktop (please complete the following information):
- OS: Ubuntu
- Version: 18.04.6 LTS (Bionic Beaver)
- Branch: v0.40.0-dev.0b4c4bf39a
- Commit hash: 0b4c4bf39a6b54f0bdaaef924e67788c35d10d6e
@mpetrunic could you comment in what situations Lodestar may throw this error? Why could the bootnode keypair wrong? I'm not familiar with noise internals
Error is decorated here
Error: Error occurred during XX handshake:
- https://github.com/ChainSafe/js-libp2p-noise/blob/1e3d20738c90d68133011a59bb5054138ee9e2b5/src/noise.ts#L195
Error occurred while verifying signed payload: one of this three places, which one?
- https://github.com/ChainSafe/js-libp2p-noise/blob/1e3d20738c90d68133011a59bb5054138ee9e2b5/src/handshake-xx.ts#L103
- https://github.com/ChainSafe/js-libp2p-noise/blob/1e3d20738c90d68133011a59bb5054138ee9e2b5/src/handshake-xx.ts#L138
- https://github.com/ChainSafe/js-libp2p-noise/blob/1e3d20738c90d68133011a59bb5054138ee9e2b5/src/handshake-xx-fallback.ts#L78
Static key doesn't match to peer that signed payload!
- https://github.com/ChainSafe/js-libp2p-noise/blob/1e3d20738c90d68133011a59bb5054138ee9e2b5/src/utils.ts#L104
Already wrote on discord couple of days ago. Pretty sure its not related to noise. Could be lodestar messing with peerId in libp2p
I've also seen this occasionally (8 times in a ~24hr period) in #4114
The stack trace seems a little more helpful now tho
2022-07-11T16:57:34.734Z libp2p:upgrader:error Failed to upgrade outbound connection UnexpectedPeerError: Error occurred during XX handshake: Error occurred while verifying signed payload: Static key doesn't match to peer that signed payload!
at XXHandshake.exchange (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/handshake-xx.ts:103:15)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Noise.performXXHandshake (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/noise.ts:220:7)
at Noise.performHandshake (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/noise.ts:165:14)
at Noise.secureOutbound (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/noise.ts:75:23)
at EventTarget._encryptOutbound (file:///home/cayman/Code/lodestar/node_modules/libp2p/src/upgrader.ts:541:12)
at EventTarget.upgradeOutbound (file:///home/cayman/Code/lodestar/node_modules/libp2p/src/upgrader.ts:248:11)
at TCP.dial (file:///home/cayman/Code/lodestar/node_modules/@libp2p/tcp/src/index.ts:37:18)
at EventTarget.dial (file:///home/cayman/Code/lodestar/node_modules/libp2p/src/transport-manager.ts:115:14) {
code: 'ERR_ENCRYPTION_FAILED'
}
also seeing hundreds / thousands of this similar-ish stack trace
2022-07-12T04:13:30.447Z libp2p:upgrader:error Failed to upgrade outbound connection UnexpectedPeerError: Error occurred during XX handshake: Error occurred while verifying signed payload: Peer ID doesn't match libp2p public key. at XXHandshake.exchange (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/handshake-xx.ts:103:15)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (node:internal/process/task_queues:96:5)
at Noise.performXXHandshake (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/noise.ts:220:7)
at Noise.performHandshake (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/noise.ts:165:14)
at Noise.secureOutbound (file:///home/cayman/Code/lodestar/node_modules/@chainsafe/libp2p-noise/src/noise.ts:75:23)
at EventTarget._encryptOutbound (file:///home/cayman/Code/lodestar/node_modules/libp2p/src/upgrader.ts:541:12)
at EventTarget.upgradeOutbound (file:///home/cayman/Code/lodestar/node_modules/libp2p/src/upgrader.ts:248:11)
at TCP.dial (file:///home/cayman/Code/lodestar/node_modules/@libp2p/tcp/src/index.ts:37:18)
at EventTarget.dial (file:///home/cayman/Code/lodestar/node_modules/libp2p/src/transport-manager.ts:115:14) {
code: 'ERR_ENCRYPTION_FAILED'
}
So this second error "Peer ID doesn't match..." happens when establishing an outbound connection, we're passing in a peer id during handshake initialization that differs from the one thats discovered during the handshake. (Maybe like dialing a discovered ENR to a node that's rebooted and generated a new peer id).
Not sure yet about the "Static key doesn't match...". It seems we create the remote static key initialized to a zeroed Uint8Array, and it should get set here https://github.com/ChainSafe/js-libp2p-noise/blob/master/src/handshakes/xx.ts#L91. I'd guess in these error cases, its not getting set there.