paloma
paloma copied to clipboard
palomachain
BUG: Report from community member that the chain appears to be struggling. Validators are missing blocks
What is happening?
Section description
Provide as much context as you can. Give as much context as you can to make it easier for the developers to figure what is happening.
3:52PM INF Timed out dur=772.764095 height=14112374 module=consensus round=0 step=1 3:52PM INF received proposal module=consensus proposal={"Type":32,"block_id":{"hash":"3FF9D3B97FB8B2C9923E80D9802F72BAFD691D20B679F0B4165638A214A9A2B5","parts":{"hash":"C020DBF45F9DD1BAC70290D5440DC2395437ACD44672BC727492073ECA20D0B9","total":1}},"height":14112374,"pol_round":-1,"round":0,"signature":"LkOpLyC6jL59oSiLmkPLfHFnAzqX8YtFaNAomun1aOh5khsAIJXG73C5un5URAQy0qLzUXDrqyqcmwTOItnNAw==","timestamp":"2024-03-10T14:52:31.442021537Z"} 3:52PM INF received complete proposal block hash=3FF9D3B97FB8B2C9923E80D9802F72BAFD691D20B679F0B4165638A214A9A2B5 height=14112374 module=consensus 3:52PM INF finalizing commit of block hash={} height=14112374 module=consensus num_txs=0 root=86A92D5C2E0DC886647BDCC39646684E20DE915BB41C7C19C7D4A7615B3A5EAF 3:52PM INF minted coins from module account amount=63265634ugrain from=mint module=x/bank 3:52PM INF abci-validator-size module=x/palomaconsensus 3:52PM ERR consensus not achieved error="evm: consensus not achieved" component=attest-router module=x/evm msg-id=449447 msg-nonce="\x00\x00\x00\x00\x00\x06ۧ" 3:52PM ERR consensus not achieved error="evm: consensus not achieved" component=attest-router module=x/evm msg-id=449456 msg-nonce="\x00\x00\x00\x00\x00\x06۰" 3:52PM ERR consensus not achieved error="evm: consensus not achieved" component=attest-router module=x/evm msg-id=449457 msg-nonce="\x00\x00\x00\x00\x00\x06۱" 3:52PM INF trying to deploy smart contract to EVM chain chain-reference-id=arbitrum-main module=x/evm smart-contract-id=6 3:52PM INF trying to deploy smart contract to EVM chain chain-reference-id=base-main module=x/evm smart-contract-id=6 3:52PM INF trying to deploy smart contract to EVM chain chain-reference-id=blast-main module=x/evm smart-contract-id=6 3:52PM INF trying to deploy smart contract to EVM chain chain-reference-id=bnb-main module=x/evm smart-contract-id=6 3:52PM INF trying to deploy smart contract to EVM chain chain-reference-id=eth-main module=x/evm smart-contract-id=6
4:01PM INF trying to deploy smart contract to EVM chain chain-reference-id=bnb-main module=x/evm smart-contract-id=6 4:01PM INF trying to deploy smart contract to EVM chain chain-reference-id=eth-main module=x/evm smart-contract-id=6 4:01PM INF trying to deploy smart contract to EVM chain chain-reference-id=gnosis-main module=x/evm smart-contract-id=6 4:01PM INF trying to deploy smart contract to EVM chain chain-reference-id=matic-main module=x/evm smart-contract-id=6 4:01PM INF trying to deploy smart contract to EVM chain chain-reference-id=op-main module=x/evm smart-contract-id=6 4:01PM INF trying to deploy smart contract to all chains module=x/evm smart-contract-id=6 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime module=x/metrix signed-blocks-window=10000 validator-conspub=palomavalcons1qsys7shahzl3fplvu0gzfz9tjqxqzz3e0vulsg 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=true missed-blocks-counter=32 module=x/metrix signed-blocks-window=10000 uptime=0.000000000000000000 validator=palomavaloper1d3v3jh6l2r23y9kgzdrahx0ev8ez0g8qj384sr validator-conspub=palomavalcons1pyjrnzmdh28ztnvk2fhhedstasfftcrk48cc46 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=true missed-blocks-counter=2 module=x/metrix signed-blocks-window=10000 uptime=0.000000000000000000 validator=palomavaloper10up3f4uxtsfyjkvdypag2q7vr7yxahzd7tex99 validator-conspub=palomavalcons1zpa3rkd0nr72qvwdlvv63tdddj6qpag95rgfnf 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=true missed-blocks-counter=130 module=x/metrix signed-blocks-window=10000 uptime=0.000000000000000000 validator=palomavaloper1rv786ey6j863szlkpwlrv784q6vgjm8438ysm3 validator-conspub=palomavalcons1gazwy9pdm34w8j44m3rn9vcfhtku9gp8g7y90s 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=true missed-blocks-counter=523 module=x/metrix signed-blocks-window=10000 uptime=0.000000000000000000 validator=palomavaloper1my3gpyx7sdx7wn4rd0hmng60q9jhykxhx3fwkh validator-conspub=palomavalcons1s73va6tzls4yy0rlevtx6dlehrfuf2f9j8689k 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=false missed-blocks-counter=12 module=x/metrix signed-blocks-window=10000 uptime=0.998800000000000000 validator=palomavaloper1z9fgzh7mzqgu33pdkxw0dqmqgm9l8exj6rl5wj validator-conspub=palomavalcons13lz359kh5eac03eqsuuysejkrh32622stz7jzq 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=false missed-blocks-counter=106 module=x/metrix signed-blocks-window=10000 uptime=0.989400000000000000 validator=palomavaloper1tdw23fpnxh2uk3djtteh7eaydymrfgnaepkpkz validator-conspub=palomavalcons15226stct8m5y58a7jq0qhh90ngppwud3cqvvtd 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=true missed-blocks-counter=16 module=x/metrix signed-blocks-window=10000 uptime=0.000000000000000000 validator=palomavaloper17kdfltcu45llx54ue0fvj8m4z9gd3ps4kcfsuq validator-conspub=palomavalcons1hum9a970ulpnx0lex9hlfww7gr6fjrem07sx7n 4:01PM ERR no validator found for cons pub address. component=metrix.UpdateUptime is-jailed=true missed-blocks-counter=16 module=x/metrix signed-blocks-window=10000 uptime=0.000000000000000000 validator=palomavaloper17kdfltcu45llx54ue0fvj8m4z9gd3ps4kcfsuq validator-conspub=palomavalcons1c0tfdeu6t7crenhr0qg78y2dda654raxfzzqne 4:01PM INF executed block height=14112690 module=state num_invalid_txs=0 num_valid_txs=2 4:01PM INF commit synced commit=436F6D6D697449447B5B3933203134372033302039352032353520313238203234322032333320323035203230352031303420393820383220313933203233203135352032333020313938203239203134203539203130203130352032333020393020343520313537203235352031333120313339203235332039315D3A4437353742327D module=server 4:01PM INF committed state app_hash=5D931E5FFF80F2E9CDCD686252C1179BE6C61D0E3B0A69E65A2D9DFF838BFD5B height=14112690 module=state num_txs=2 4:01PM INF Timed out dur=-289.895882 height=14112691 module=consensus round=0 step=1 4:01PM INF indexed block exents height=14112690 module=txindex
@verabehr @byte-bandit Yup, VolumeFi just missed a bunch of blocks as well in the past few minutes. I wonder this is blast issue?
I don't think this has anything to do with the BLAST integration, there was no code change to support the chain. This might have been going on for a while.
Looking at the logs, I cannot find any obvious reason. From what I can tell, validators sign and commit their blocks just fine. However, looking at the data above, the average block time was 1.15 seconds, whereas the average should be roughly 1.6s at the moment.
It's possible that this is simply too fast. A lot of the features on chain have not yet been performance optimised, so some slower machines might not be able to keep up.
That being said, I sometimes see our validator fail on blocks with only 2 transactions in them. Still, you have network overhead, IO, etc... to deal with as well, which will add a couple 100 ms already. I'm not sure how long the window stays open after a new block has been proposed, but it's possible some validators just won't make it before 2/3 consensus is reached and the window closes?
The errors observed in the logs above shouldn't affect block execution. The metrix errors are literally fire & forget without consequence, but I should fix these. The consensus ones are expected as well as messages are examined for their attestation. I don't see how they would prevent some validators from missing blocks.
The best idea I have would be to do some profiling and check what exactly is happening.
