node-binary icon indicating copy to clipboard operation
node-binary copied to clipboard

Published 20 hours ago • verified publisher icon bnb-chain

Error on syncing fullnode

Open astudnev opened this issue 6 years ago • 19 comments

I got the following error on block ~ 16 million: ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash. Expected 2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B, got 937156DCB2220A0B323B4CD7040C368765F86DC76399090579AA37DB945DB66E

bnc.log shows:

I[2019-10-01|20:47:14.399] Added Order module=dex order="NewOrderMsg{Sender: 34AB7C199991F2A892E1D27E072D0C77AA3601A4, Id: 34AB7C199991F2A892E1D27E072D0C77AA3601A4-230, Symbol: LTO-BDF_BNB}" I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770239 I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770240 I[2019-10-01|20:47:14.399] Added Order module=dex order="NewOrderMsg{Sender: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D, Id: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D-356, Symbol: PHB-2DF_BNB}" I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770240 I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770241 I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770241 I[2019-10-01|20:47:14.399] No new orders for any pair, give up matching module=dexkeeper I[2019-10-01|20:47:14.399] No order comes in for the block module=dexkeeper I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770242 I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770242 I[2019-10-01|20:47:14.399] No new orders for any pair, give up matching module=dexkeeper I[2019-10-01|20:47:14.399] No order comes in for the block module=dexkeeper I[2019-10-01|20:47:14.399] Relaying block for order book module=dex height=16770243 I[2019-10-01|20:47:14.399] does not recover invalid tx module=dex txHash=54D51EFAD447F13CFEF76695D1CF48B39B7481B729836F025A27234155A84036 err="{"codespace":1,"code":1,"abci_code":65537,"message":"recovered: open /home/studnev/.bnbchaind/data/application.db/1349323.ldb: too many open files\nstack:\ngoroutine 39571 [running]:\nruntime/debug.Stack(0xc0014265c0, 0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc001427aa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:821 +0x6e\npanic(0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc0004c4358, 0xc00af8e0c0, 0x29, 0x30, 0x29, 0xc00af8e0c0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc000bda660, 0xc00af8e090, 0x21, 0x21, 0x0, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc000b99810, 0xc006b1c120, 0x20, 0x20, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:80 +0x29e\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0a50, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014268f0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0370, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426950, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb02c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014269b0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0210, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a10, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644dc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a70, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644d10, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426ad0, 0xc001426b90)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00853f1e0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b30, 0xc001426bf0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606cbb0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b90, 0xc001426c50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c9a0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426bf0, 0xc001426cb0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c8f0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426c50, 0xc001426d10)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00817cdc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426cb0, 0xc001426d70)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e7ad0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426d10, 0xc001426dd0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e78c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x7fe6ed37c008, 0x41a309)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00bfd7760, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x10000000043b5f1, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00ce57340, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc014429690, 0xc001426e88)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012685080, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x20, 0x103a880, 0xc0126a9ca0, 0xc0126a9ca0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc002427760, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xc0126a9ca0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:153 +0x86\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc001c7a820, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x1026560, 0xc001427060)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0x113a200, 0x15001a0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0xc005dda800, 0x14b49b8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\ngithub.com/binance-chain/node/common/tx.processAccount(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:233 +0xc2\ngithub.com/binance-chain/node/common/tx.NewAnteHandler.func1(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14d9a20, 0xc009872840, 0x1026505, 0xc014429670, 0x14f7580, 0xc017eab260, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:151 +0x599\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc0002ede00, 0x11fb905, 0x14d9a20, 0xc009872840, 0xc00a6c7840, 0x40, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:834 +0x6e3\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc0002ede00, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:657 +0x270\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0004d8500, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/app/app.go:440 +0x9b\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc000b10a50, 0xc001427fa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc000b10a50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\n"}" height=16770243 I[2019-10-01|20:47:14.399] replayed all tx. Starting match module=dex height=16770243 I[2019-10-01|20:47:14.399] No new orders for any pair, give up matching module=dexkeeper I[2019-10-01|20:47:14.399] No order comes in for the block module=dexkeeper I[2019-10-01|20:47:14.638] Located day to load breathe block height module=dexkeeper epochDay="\ufffdF\u0000\u0000\u0000\u0000\u0000\u0000" I[2019-10-01|20:47:14.638] Loaded breathe block height module=dexkeeper height=16689772 I[2019-10-01|20:47:14.638] get last breathe block height module=main height=16689772 I[2019-10-01|20:47:14.645] reload latest snapshot module=statesync height=16689772 I[2019-10-01|20:47:14.836] ABCI Handshake App Info module=consensus height=16770243 hash=2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B software-version= protocol-version=0 I[2019-10-01|20:47:14.838] ABCI Replay Blocks module=consensus appHeight=16770243 storeHeight=16770244 stateHeight=16770243 I[2019-10-01|20:47:14.838] Replay last block using real app module=consensus

astudnev avatar Oct 01 '19 20:10 astudnev

version 0.6.1. Ubuntu OS

astudnev avatar Oct 01 '19 20:10 astudnev

Problem is that even after extending ulimit it does not recover

astudnev avatar Oct 01 '19 21:10 astudnev

@astudnev How much limit did you set?

What's the new error after you restart?

chainwhisper avatar Oct 02 '19 13:10 chainwhisper

I set ulimit 65535 and after restart getting the same error as i posted

astudnev avatar Oct 02 '19 13:10 astudnev

@astudnev

Hello, can you please share the output for ulimit -a

chainwhisper avatar Oct 03 '19 13:10 chainwhisper

@astudnev what is your log now after you set ulimit 65535 , can you post your app.toml as well?

unclezoro avatar Oct 03 '19 13:10 unclezoro

ulimit -a core file size (blocks, -c) 0 data seg size (kbytes, -d) unlimited scheduling priority (-e) 0 file size (blocks, -f) unlimited pending signals (-i) 256696 max locked memory (kbytes, -l) 64 max memory size (kbytes, -m) unlimited open files (-n) 65535 pipe size (512 bytes, -p) 8 POSIX message queues (bytes, -q) 819200 real-time priority (-r) 0 stack size (kbytes, -s) 8192 cpu time (seconds, -t) unlimited max user processes (-u) 256696 virtual memory (kbytes, -v) unlimited file locks (-x) unlimited

astudnev avatar Oct 03 '19 18:10 astudnev

./linux/bnbchaind --home ~/.bnbchaind start ERROR: Error during handshake: Error on replay: Wrong Block.Header.AppHash. Expected 2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B, got 937156DCB2220A0B323B4CD7040C368765F86DC76399090579AA37DB945DB66E

astudnev avatar Oct 03 '19 18:10 astudnev

bnc.log tail

I[2019-10-03|18:53:48.833] replayed all tx. Starting match module=dex height=16770236 I[2019-10-03|18:53:48.833] No new orders for any pair, give up matching module=dexkeeper I[2019-10-03|18:53:48.833] No order comes in for the block module=dexkeeper I[2019-10-03|18:53:48.833] Relaying block for order book module=dex height=16770237 I[2019-10-03|18:53:48.833] replayed all tx. Starting match module=dex height=16770237 I[2019-10-03|18:53:48.833] No new orders for any pair, give up matching module=dexkeeper I[2019-10-03|18:53:48.833] No order comes in for the block module=dexkeeper I[2019-10-03|18:53:48.833] Relaying block for order book module=dex height=16770238 I[2019-10-03|18:53:48.833] replayed all tx. Starting match module=dex height=16770238 I[2019-10-03|18:53:48.833] No new orders for any pair, give up matching module=dexkeeper I[2019-10-03|18:53:48.833] No order comes in for the block module=dexkeeper I[2019-10-03|18:53:48.833] Relaying block for order book module=dex height=16770239 I[2019-10-03|18:53:48.834] Added Order module=dex order="NewOrderMsg{Sender: 34AB7C199991F2A892E1D27E072D0C77AA3601A4, Id: 34AB7C199991F2A892E1D27E072D0C77AA3601A4-230, Symbol: LTO-BDF_BNB}" I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770239 I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770240 I[2019-10-03|18:53:48.834] Added Order module=dex order="NewOrderMsg{Sender: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D, Id: 4792B6F9891BD0B34235DC0F177C14EA21D4D23D-356, Symbol: PHB-2DF_BNB}" I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770240 I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770241 I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770241 I[2019-10-03|18:53:48.834] No new orders for any pair, give up matching module=dexkeeper I[2019-10-03|18:53:48.834] No order comes in for the block module=dexkeeper I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770242 I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770242 I[2019-10-03|18:53:48.834] No new orders for any pair, give up matching module=dexkeeper I[2019-10-03|18:53:48.834] No order comes in for the block module=dexkeeper I[2019-10-03|18:53:48.834] Relaying block for order book module=dex height=16770243 I[2019-10-03|18:53:48.834] does not recover invalid tx module=dex txHash=54D51EFAD447F13CFEF76695D1CF48B39B7481B729836F025A27234155A84036 err="{"codespace":1,"code":1,"abci_code":65537,"message":"recovered: open /home/studnev/.bnbchaind/data/application.db/1349323.ldb: too many open files\nstack:\ngoroutine 39571 [running]:\nruntime/debug.Stack(0xc0014265c0, 0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/debug/stack.go:24 +0x9d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx.func1(0xc001427aa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:821 +0x6e\npanic(0x109c540, 0xc00421c000)\n\t/usr/local/go/src/runtime/panic.go:522 +0x1b5\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*GoLevelDB).Get(0xc0004c4358, 0xc00af8e0c0, 0x29, 0x30, 0x29, 0xc00af8e0c0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/go_level_db.go:58 +0x137\ngithub.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db.(*prefixDB).Get(0xc000bda660, 0xc00af8e090, 0x21, 0x21, 0x0, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/tendermint/libs/db/prefix_db.go:60 +0x162\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*nodeDB).GetNode(0xc000b99810, 0xc006b1c120, 0x20, 0x20, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/nodedb.go:80 +0x29e\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).getRightNode(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:363\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0a50, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014268f0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:176 +0x319\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0370, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426950, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb02c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc0014269b0, 0x65)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc009cb0210, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a10, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644dc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426a70, 0x66)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012644d10, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426ad0, 0xc001426b90)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00853f1e0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b30, 0xc001426bf0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606cbb0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426b90, 0xc001426c50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c9a0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426bf0, 0xc001426cb0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00606c8f0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426c50, 0xc001426d10)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00817cdc0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc001426cb0, 0xc001426d70)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e7ad0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xffffffffffffffff, 0xc001426d10, 0xc001426dd0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00c5e78c0, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x7fe6ed37c008, 0x41a309)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:174 +0x1aa\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00bfd7760, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0x10000000043b5f1, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc00ce57340, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0x1, 0xc014429690, 0xc001426e88)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*Node).get(0xc012685080, 0xc008b8c320, 0xc0126a9ca0, 0x1c, 0x20, 0x20, 0x103a880, 0xc0126a9ca0, 0xc0126a9ca0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/node.go:177 +0x293\ngithub.com/binance-chain/node/vendor/github.com/tendermint/iavl.(*ImmutableTree).Get(...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/tendermint/iavl/immutable_tree.go:98\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store.(*IavlStore).Get(0xc002427760, 0xc0126a9ca0, 0x1c, 0x20, 0x1c, 0xc0126a9ca0, 0x8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/store/iavlstore.go:153 +0x86\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountStoreCache).GetAccount(0xc001c7a820, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:236 +0x13d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x1026560, 0xc001427060)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda580, 0xc0126a9960, 0x14, 0x14, 0x0, 0x0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).getAccountFromCache(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0x113a200, 0x15001a0)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:359 +0x14d\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.(*accountCache).GetAccount(0xc005dda800, 0xc0126a9960, 0x14, 0x14, 0xc005dda800, 0x14b49b8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:299 +0x49\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth.AccountKeeper.GetAccount(0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/x/auth/mapper.go:87 +0xcb\ngithub.com/binance-chain/node/common/tx.processAccount(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14e86c0, 0xc000076db0, 0x136fe68, 0xc0004cfab0, 0xc0126a9960, 0x14, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:233 +0xc2\ngithub.com/binance-chain/node/common/tx.NewAnteHandler.func1(0x14f7580, 0xc017eab260, 0xc005dda600, 0x10, 0x14d9a20, 0xc009872840, 0x1026505, 0xc014429670, 0x14f7580, 0xc017eab260, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/common/tx/ante.go:151 +0x599\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).RunTx(0xc0002ede00, 0x11fb905, 0x14d9a20, 0xc009872840, 0xc00a6c7840, 0x40, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:834 +0x6e3\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp.(*BaseApp).DeliverTx(0xc0002ede00, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/baseapp/baseapp.go:657 +0x270\ngithub.com/binance-chain/node/app.(*BinanceChain).DeliverTx(0xc0004d8500, 0xc004576ff0, 0xe1, 0xe1, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)\n\t/home/suyu/go/src/github.com/binance-chain/node/app/app.go:440 +0x9b\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker.func1(0xc000b10a50, 0xc001427fa8)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:142 +0x1b6\ngithub.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).deliverTxWorker(0xc000b10a50)\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:151 +0x8c\ncreated by github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent.(*asyncLocalClient).OnStart\n\t/home/suyu/go/src/github.com/binance-chain/node/vendor/github.com/cosmos/cosmos-sdk/server/concurrent/async_local_client.go:93 +0x64\n"}" height=16770243 I[2019-10-03|18:53:48.834] replayed all tx. Starting match module=dex height=16770243 I[2019-10-03|18:53:48.834] No new orders for any pair, give up matching module=dexkeeper I[2019-10-03|18:53:48.834] No order comes in for the block module=dexkeeper I[2019-10-03|18:53:49.070] Located day to load breathe block height module=dexkeeper epochDay="\ufffdF\u0000\u0000\u0000\u0000\u0000\u0000" I[2019-10-03|18:53:49.070] Loaded breathe block height module=dexkeeper height=16689772 I[2019-10-03|18:53:49.070] get last breathe block height module=main height=16689772 I[2019-10-03|18:53:49.077] reload latest snapshot module=statesync height=16689772 I[2019-10-03|18:53:49.266] ABCI Handshake App Info module=consensus height=16770243 hash=2902E81E5C1C9A789471CAF607CDCC2747A97A6C3257C9FA42D361A8D5E01D8B software-version= protocol-version=0 I[2019-10-03|18:53:49.273] ABCI Replay Blocks module=consensus appHeight=16770243 storeHeight=16770244 stateHeight=16770243 I[2019-10-03|18:53:49.273] Replay last block using real app module=consensus

astudnev avatar Oct 03 '19 18:10 astudnev

it seems you are on mainnet. Which version are you using?

chainwhisper avatar Oct 04 '19 01:10 chainwhisper

0.6.1

astudnev avatar Oct 05 '19 05:10 astudnev

0.6.1

can you please share your app.toml file please?

chainwhisper avatar Oct 05 '19 12:10 chainwhisper 

cat .bnbchaind/config/app.toml 
# This is a TOML config file.
# For more information, see https://github.com/toml-lang/toml

[base]
# Interval blocks of breathe block, if breatheBlockInterval is 0, breathe block will be created every day.
breatheBlockInterval = 0
# Size of account cache
accountCacheSize = 30000
# Size of signature cache
signatureCacheSize = 30000
# Running mode when start up, 0: Normal, 1: TransferOnly, 2: RecoverOnly
startMode = 0
# Concurrency of OrderKeeper, should be power of 2
orderKeeperConcurrency = 2
# Days count back for breathe block
breatheBlockDaysCountBack = 7

[upgrade]
# Block height of BEP6 upgrade
BEP6Height = 20300000
# Block height of BEP9 upgrade
BEP9Height = 20300000
# Block height of BEP10 upgrade
BEP10Height = 20300000
# Block height of BEP19 upgrade
BEP19Height = 20300000
# Block height of BEP12 upgrade
BEP12Height = 25120000

[addr]
# Bech32PrefixAccAddr defines the Bech32 prefix of an account's address
bech32PrefixAccAddr = "bnb"
# Bech32PrefixAccPub defines the Bech32 prefix of an account's public key
bech32PrefixAccPub = "bnbp"
# Bech32PrefixValAddr defines the Bech32 prefix of a validator's operator address
bech32PrefixValAddr = "bva"
# Bech32PrefixValPub defines the Bech32 prefix of a validator's operator public key
bech32PrefixValPub = "bvap"
# Bech32PrefixConsAddr defines the Bech32 prefix of a consensus node address
bech32PrefixConsAddr = "bca"
# Bech32PrefixConsPub defines the Bech32 prefix of a consensus node public key
bech32PrefixConsPub = "bcap"

##### publication related configurations #####
[publication]
# configurations ends with Kafka can be a semi-colon separated host-port list
# Whether we want publish market data (this includes trades and order)
publishOrderUpdates = true
orderUpdatesTopic = "orders"
orderUpdatesKafka = "127.0.0.1:9092"

# Whether we want publish account balance to notify browser db indexer persist latest account balance change
publishAccountBalance = true
accountBalanceTopic = "accounts"
accountBalanceKafka = "127.0.0.1:9092"

# Whether we want publish order book changes
publishOrderBook = true
orderBookTopic = "orders"
orderBookKafka = "127.0.0.1:9092"

# Whether we want publish block fee changes
publishBlockFee = true
blockFeeTopic = "accounts"
blockFeeKafka = "127.0.0.1:9092"

# Whether we want publish transfers
publishTransfer = true
transferTopic = "transfers"
transferKafka = "127.0.0.1:9092"

# Global setting
publicationChannelSize = "10000"
publishKafka = false
publishLocal = true



# max size in megabytes of marketdata json file before rotate
localMaxSize = 1024
# max days of marketdata json files to keep before deleted
localMaxAge = 7000

[log]

# Write logs to console instead of file
logToConsole = false

## The below parameters take effect only when logToConsole is false
# Log file root, if not set, use home path
logFileRoot = ""
# Log file path relative to log file root path
logFilePath = "bnc.log"
# Number of logs keep in memory before writing to file
logBuffSize = 10000`

astudnev avatar Oct 05 '19 15:10 astudnev

@astudnev Thks for report the issue. I think it is a potential bug for binance chain. The reason is that at first the machine that run bnbchaind do not set large enough open files, and cause one Transaction failed(which is supposed to be successed). As I see, your fullnode have synced 16770243 blocks while the chain is in 43159598, you can clean the data and use state-sync(will help you quick catch up). Please refer to https://docs.binance.org/fullnode.html. As you have configed 65535 max open files, it won't happen again.

unclezoro avatar Oct 07 '19 04:10 unclezoro

I think we should backup data folder first, then use https://github.com/binance-chain/node-binary/tree/master/tools/recover tool to rollback to height 16770242. then restart

chainwhisper avatar Oct 07 '19 05:10 chainwhisper

OK, thank you, i already synced again.

astudnev avatar Oct 07 '19 06:10 astudnev

OK, thank you, i already synced again.

Great!

Did you re-sync from the start?

chainwhisper avatar Oct 08 '19 03:10 chainwhisper

OK, thank you, i already synced again.

Great!

Did you re-sync from the start?

yes

astudnev avatar Oct 08 '19 04:10 astudnev

I also tried recover tool, it worked fine in this situation

astudnev avatar Oct 14 '19 10:10 astudnev