panic: Failed to process committed block: Wrong Block.Header.AppHash.

[dingyo777]

I've started new full node-binary.

spec is like below

• ubuntu 18.04
• full node version : 0.6.3-hotfix
• I copied related config : app.toml, config.toml, genesis.json

But I got below error at around block number 1082013

related logs are like below

I[2020-01-06|22:47:04.484] Executed block                               module=state height=1082013 validTxs=0 invalidTxs=1
...
panic: Failed to process committed block (1082014:39C2F09916BD3415C40BD825406C09A022C98FFCDBDFFFC7DADA0E822F60F6B9): Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

May I get your favor how to resolve this issue?

[chainwhisper]

Did you use state-sync or fast-sync?

The default way is using state-sync and if your node is syncing in mainnet, you should start from height round 59MM. It will take less than an hour to get synced.

[dingyo777]

@huangsuyu Thank you always for your support~!!

currently, my full node is fast-sync

What I want to is download all the block info from block number 0.

Then In that case, what should I config?

My full node-binary config is like this:

fast_sync = true
state_sync_reactor = false
state_sync_height = -1
hot_sync = true
hot_sync_timeout = "3s"
hot_sync_reactor = true
...

[chainwhisper]

Hello, @dingyo777

I have tested fast-sync in mainnet with my machine and it worked well.

Are you using the same config files from v0.6.3-hf?

[dingyo777]

Hello ~ @huangsuyu

Thank you for your confirmation.

Yes, I have same config from v0.6.3-hf and I modified some filed as mentioned above comment.

Dose your node work well including block number 1082013?

My node also works well until block number 1082013, but when while syncing 1082013, it failed.

[chainwhisper]

@dingyo777 yes, I used fast-sync and there was no issue.

My advice is you reset your node and start again.

[ArseniiPetrovich]

I think we run into this issue as well. Logs are a bit different but still. Logs from our node:

Jan 22 05:21:54 ip-172-31-18-149 bnbchaind: ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 377FA176F7DD9CB5F11318DD9497C239D1D93636D9DCB842DAB4C5CB7AA347B6, got F9B249154ECD284A1CA4781DFFF0EBCA883A6E8E2BAC494C312731880B42F7EE

At the same time only 1 of our 2 nodes failed, second one is still active. Any idea of what could be wrong?

[pablofullana]

hi everyone, any updates on @ArseniiPetrovich issue?

[chainwhisper]

Hello, @ArseniiPetrovich could you please set your log level to debug and share more info here?

I cannot reproduce in my end

[dingyo777]

very very weird.

I reinstall 0.6.3-hotfix on another linux ubuntu machine (18.04) I copied all the related config under ~/.bnbchaind/config

At the same time, I reinstalled 0.6.3-hotfix on original server.

But both stopped at block number 1082013 with same error like

panic: Failed to process committed block (1082014:39C2F09916BD3415C40BD825406C09A022C98FFCDBDFFFC7DADA0E822F60F6B9): Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

[chainwhisper]

very very weird.

I reinstall 0.6.3-hotfix on another linux ubuntu machine (18.04) I copied all the related config under ~/.bnbchaind/config

At the same time, I reinstalled 0.6.3-hotfix on original server.

But both stopped at block number 1082013 with same error like

panic: Failed to process committed block (1082014:39C2F09916BD3415C40BD825406C09A022C98FFCDBDFFFC7DADA0E822F60F6B9): Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

can you please change log level to debug and share more details?

Thanks!

[ArseniiPetrovich]

Hey, @huangsuyu, not sure if I'm doing it correct, but here the output:

[root@ip-172-31-18-149 ~]# /home/ec2-user/node-binary/fullnode/prod/0.6.3-hotfix/linux/bnbchaind start --home=/home/ec2-user/.bnbchaind --log_level *:info --trace
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected 377FA176F7DD9CB5F11318DD9497C239D1D93636D9DCB842DAB4C5CB7AA347B6, got F9B249154ECD284A1CA4781DFFF0EBCA883A6E8E2BAC494C312731880B42F7EE

I think I should get a bit more verbose output, but that is all I got.

[ArseniiPetrovich]

Oh, sorry. I forgot about bnc.log file. Here it is - it was too big to paste it here, so I've uploaded it to my GDrive. https://drive.google.com/file/d/1TrF3TWvUiCcn_RCxF5LkYX9As45rA6lN/view?usp=sharing ZIP file format.

[dingyo777]

@huangsuyu Thank you always~! I changed log level to debug at config.toml but nothing special. Below is some part of messages right before die

D[2020-02-04|02:02:52.154] Got recentPrices is only debug-level log.

I[2020-02-04|02:02:52.153] Relaying block for order book                module=dex height=1082011
I[2020-02-04|02:02:52.153] replayed all tx. Starting match              module=dex height=1082011
I[2020-02-04|02:02:52.153] No new orders for any pair, give up matching module=dexkeeper
I[2020-02-04|02:02:52.153] No order comes in for the block              module=dexkeeper
I[2020-02-04|02:02:52.153] Relaying block for order book                module=dex height=1082012
I[2020-02-04|02:02:52.153] replayed all tx. Starting match              module=dex height=1082012
I[2020-02-04|02:02:52.153] No new orders for any pair, give up matching module=dexkeeper
I[2020-02-04|02:02:52.153] No order comes in for the block              module=dexkeeper
I[2020-02-04|02:02:52.153] Relaying block for order book                module=dex height=1082013
I[2020-02-04|02:02:52.153] Skip tx when replay                          module=dex height=1082013 idx=0
I[2020-02-04|02:02:52.153] replayed all tx. Starting match              module=dex height=1082013
I[2020-02-04|02:02:52.153] No new orders for any pair, give up matching module=dexkeeper
I[2020-02-04|02:02:52.153] No order comes in for the block              module=dexkeeper
D[2020-02-04|02:02:52.154] Got recentPrices                             module=main lastSeq=1081 recentPrices=map[]
I[2020-02-04|02:02:52.217] Starting multiAppConn                        module=proxy impl=multiAppConn
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=query impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=state impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=mempool impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting asyncLocalClient                    module=abci-client connection=consensus impl=asyncLocalClient
I[2020-02-04|02:02:52.217] Starting EventBus                            module=events impl=EventBus
I[2020-02-04|02:02:52.217] Starting PubSub                              module=pubsub impl=PubSub
I[2020-02-04|02:02:52.234] Starting TxIndexerService                    module=txindex impl=TxIndexerService
I[2020-02-04|02:02:52.234] Starting BlockIndexerService                 module=blockindex impl=BlockIndexerService
I[2020-02-04|02:02:52.235] Starting indexHub                            module=indexer_hub impl=indexHub
I[2020-02-04|02:02:52.235] ABCI Handshake App Info                      module=consensus height=1082013 hash=AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0 software-version= protocol-version=0
I[2020-02-04|02:02:52.235] ABCI Replay Blocks                           module=consensus appHeight=1082013 storeHeight=1082014 stateHeight=1082013
I[2020-02-04|02:02:52.235] Replay last block using real app             module=consensus
ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash.  Expected AC9FAC16513AF96A38822F2DF9276131D6ABDCD9EF47A6EEDACF5CA3E54883C0, got 92EB0FF45ED3F1EDD8A30829C2712EF5DFA7F09A2F83B24AC330A2BF5B14BE9E

---

Above logs are made when I retried on failed node-binary.

So let me start again after resetting all the data file and related config file with debug level. And then I will attach the result here

[chainwhisper]

hello, @dingyo777

I failed to reproduce your issue at my end. I have enabled fast-sync and downloaded all the blocks.

my config.toml:

fast_sync = true

# As state sync is an experimental feature, this switch can totally disable it on core network nodes (validator, witness)
state_sync_reactor = true

# If this node is many days behind the tip of the chain, StateSyncHeight
# allows them to catchup quickly by downloading app state (without historical blocks)
# in parallel and start syncing block afterwards
# <0 - turn off state sync
# =0 - sync from peer's latest height
# >0 - sync from that height
state_sync_height = -1

# Only take effect when HotSyncReactor is true.
# If true, will sync blocks use hot sync protocol
# If false, still use tendermint consensus protocol, but can still handle other peers sync request.
hot_sync = true

# The max wait time for subscribe a block.
# Only take effect when hot_sync is true
hot_sync_timeout = "3s"

# It will benefit fullnode and witness who do not need consensus by saving network and cpu resources.
# Recommend the node that is not validator to turn on.
hot_sync_reactor = true

[dingyo777]

@huangsuyu Really thank you so much for your support~!! Then let me retry with the config you attached.

[ArseniiPetrovich]

What about my issue, @huangsuyu ?

[chainwhisper]

What about my issue, @huangsuyu ?

please share your config.toml and app.toml.

I failed to reproduce your problem at my end.

[ArseniiPetrovich]

Sure, here it is, @huangsuyu config_and_app_toml.zip

[ares0x]

Hi, I ran into a problem. My blocks were syncing normally the other day. Today I saw that he made a mistake.

version:0.6.3

panic: github.com/tendermint/tendermint/blockchain/hot.(*BlockPool).applyBlock(0xc000e82000, 0xc0fc88ed80) /home/suyu/go/pkg/mod/github.com/binance-chain/[email protected]/blockchain/hot/pool.go:566 +0x494 github.com/tendermint/tendermint/blockchain/hot.(*BlockPool).hotSyncRoutine(0xc000e82000) /home/suyu/go/pkg/mod/github.com/binance-chain/[email protected]/blockchain/hot/pool.go:383 +0x3c6 created by github.com/tendermint/tendermint/blockchain/hot.(*BlockPool).SwitchToHotSync /home/suyu/go/pkg/mod/github.com/binance-chain/[email protected]/blockchain/hot/pool.go:198 +0x172 ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B ERROR: error during handshake: error on replay: Wrong Block.Header.AppHash. Expected F678ABFD3A8AE513A7DFE1BC58E94341C743476E652CE18A77D82B7F3AA52382, got 8B86293673C78CC4B001122A09717602A48F7A8CB210DCBBEAAC6EBA1633368B

@huangsuyu

config: fast_sync = true state_sync_reactor = true state_sync_height = 0 hot_sync = true hot_sync_timeout = "3s" hot_sync_reactor = true

[chainwhisper]

Hello, @404killer

Could you upgrade to hotfix version v0.6.3-hf1? you can use this script

[ares0x]

Can I use this script to upgrade the version of the node directly?

Hello, @404killer

Could you upgrade to hotfix version v0.6.3-hf1? you can use this script

Can I use this script to upgrade the version of the node directly?

[ares0x]

Can I use this script to upgrade the version of the node directly?

Hello, @404killer Could you upgrade to hotfix version v0.6.3-hf1? you can use this script

Can I use this script to upgrade the version of the node directly?

And do I need to resynchronize my block data?

[ares0x]

`... Choose version of Binance Chain node to install

1. 0.5.8
2. 0.5.9
3. 0.5.10
4. 0.6.0
5. 0.6.1
6. 0.6.2
7. 0.6.3 `

My current version is 0.6.3. There is no v0.6.3-hf1

[chainwhisper]

`... Choose version of Binance Chain node to install

1. 0.5.8
2. 0.5.9
3. 0.5.10
4. 0.6.0
5. 0.6.1
6. 0.6.2
7. 0.6.3 `

My current version is 0.6.3. There is no v0.6.3-hf1

This script is already updated, please download the latest version.

To recover, please follow this instruction here: https://docs.binance.org/fullnodeissue.html#forget-to-upgrade

You need to use recovery tool to revert one block and restart with hotfix version.

[ares0x]

`... Choose version of Binance Chain node to install

1. 0.5.8
2. 0.5.9
3. 0.5.10
4. 0.6.0
5. 0.6.1
6. 0.6.2
7. 0.6.3 `

My current version is 0.6.3. There is no v0.6.3-hf1

This script is already updated, please download the latest version.

To recover, please follow this instruction here: https://docs.binance.org/fullnodeissue.html#forget-to-upgrade

You need to use recovery tool to revert one block and restart with hotfix version.

Ok, I'll test it first. Thank you very much

[wannabit-yongjoo]

Checked "state_recover" working well.

Used like state_recover 1181875 $BNCHOME

[ares0x]

I upgraded the version of node-binary to 0.6.3-hotfix to fix the problem.And thank you very much for your help. @huangsuyu

[kenorb]

See: https://github.com/cosmos/cosmos-sdk/issues/10281