theta-protocol-ledger
theta-protocol-ledger copied to clipboard
thetatoken
Fail after update
I 've got the following, after updating to 3.4.0 and restart my node.
[2022-09-15 10:34:38] INFO [snapshot] Snapshot loaded successfully.
[2022-09-15 10:34:38] DEBUG [ledger] Reseting state to height 14760131, hash 0xaba8e95b98bc6f1555c84b304b6b5e853cc6fd19d34abd4dee9345a5be6c4e0b
[2022-09-15 10:34:38] INFO [p2p] Connecting to seed and persisted peers...
[2022-09-15 10:34:38] WARN [p2p] Failed to retrieve previously persisted peers
[2022-09-15 10:34:38] DEBUG [consensus] Enter epoch 14845755
[2022-09-15 10:34:38] DEBUG [consensus] shouldPropose=false: epoch is behind epoch=14845755 tip=0x6062f42237d7ab0021ebe7df7e98c9e43b14579406dfce2ff009b9fefd800c38 tip.Epoch=14845755
[2022-09-15 10:34:38] INFO [p2p] InboundPeerListener listen routine started, seedPeerOnly set to false
[2022-09-15 10:34:38] DEBUG peers is : %v, stringbuilder is : %s
[]
[2022-09-15 10:34:38] DEBUG [request] refreshCounter: 0
[2022-09-15 10:34:38] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 10:34:38] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x6062f42237d7ab0021ebe7df7e98c9e43b14579406dfce2ff009b9fefd800c38]
[2022-09-15 10:34:38] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 10:34:38] WARN [reporter] Reporter failed to create request: Post http://guardian-metrics.thetatoken.org:9000/peers/set: dial tcp 34.220.181.170:9000: connect: connection refused
[2022-09-15 10:34:44] DEBUG [request] refreshCounter: 1
[2022-09-15 10:34:44] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 10:34:44] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x6062f42237d7ab0021ebe7df7e98c9e43b14579406dfce2ff009b9fefd800c38]
[2022-09-15 10:34:44] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 10:34:50] DEBUG [consensus] Epoch timeout. Repeating epoch e.epoch=14845755
[2022-09-15 10:34:50] ERROR Failed to create tree store for: 0x3ef8fcf3d1e495467b0ca0e53e35868a2db42bbcb385e051699e5490c4935471: missing trie node 3ef8fcf3d1e495467b0ca0e53e35868a2db42bbcb385e051699e5490c4935471 (path )
[2022-09-15 10:34:50] PANIC [ledger] Failed to load state for validator pool block.BlockHeader.StateHash=0x3ef8fcf3d1e495467b0ca0e53e35868a2db42bbcb385e051699e5490c4935471 block.HCC.BlockHash=0x11b729ab9262339575b163a24d212a1e3531318828028b7d07c49ce92339e081 block.Hash=0x7dad8c2d7f38b43c5cc3329e9ac076dfdbba72a4f069c0001fbd643e99a69841 block.Height=14760129 block.Status.IsTrusted()=false
panic: (*logrus.Entry) (0x12bb840,0xc0049e96c0)
goroutine 117 [running]:
github.com/sirupsen/logrus.Entry.log(0xc00008a180, 0xc01fb54e40, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:239 +0x2cf
github.com/sirupsen/logrus.(*Entry).Log(0xc0049e9650, 0x0, 0xc00b66d050, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:268 +0xc8
github.com/sirupsen/logrus.(*Entry).Panic(0xc0049e9650, 0xc00b66d050, 0x1, 0x1)
/go/pkg/mod/github.com/sirupsen/[email protected]/entry.go:306 +0x55
github.com/thetatoken/theta/ledger.(*Ledger).GetFinalizedValidatorCandidatePool(0xc00015f800, 0x3cb4387f2d8cad7d, 0xdf76c09a9e32c35c, 0xc069f0a472badb, 0x4198a6993e64bd1f, 0xc0231b0400, 0x0, 0x20, 0x20)
/go/src/github.com/thetatoken/theta/ledger/ledger.go:137 +0x6d5
github.com/thetatoken/theta/consensus.selectTopStakeHoldersAsValidatorsForBlock(0x16441e0, 0xc00ba5e180, 0xabd73722f46260, 0xe4c9987edfe7eb21, 0x2fcedf069457143b, 0x380c80fdfeb909f0, 0xc0231b0400, 0x20)
/go/src/github.com/thetatoken/theta/consensus/validator.go:161 +0x83
github.com/thetatoken/theta/consensus.(*RotatingValidatorManager).GetValidatorSet(0xc0001d5ae0, 0xabd73722f46260, 0xe4c9987edfe7eb21, 0x2fcedf069457143b, 0x380c80fdfeb909f0, 0xa01a9e37efce6e6a)
/go/src/github.com/thetatoken/theta/consensus/validator.go:128 +0x50
github.com/thetatoken/theta/consensus.(*ConsensusEngine).shouldVoteByID(0xc00ba5e180, 0xefce6e6a472f3cfe, 0x6790e9bda01a9e37, 0x22f462602a1f9943, 0xdfe7eb2100abd737, 0x9457143be4c9987e, 0xfeb909f02fcedf06, 0x2fcedf06380c80fd, 0x380c80fdfeb909f0)
/go/src/github.com/thetatoken/theta/consensus/engine.go:827 +0x4b
github.com/thetatoken/theta/consensus.(*ConsensusEngine).shouldVote(0xc00ba5e180, 0xabd73722f46260, 0xe4c9987edfe7eb21, 0x2fcedf069457143b, 0x380c80fdfeb909f0, 0x0)
/go/src/github.com/thetatoken/theta/consensus/engine.go:823 +0x9d
github.com/thetatoken/theta/consensus.(*ConsensusEngine).vote(0xc00ba5e180)
/go/src/github.com/thetatoken/theta/consensus/engine.go:835 +0xd7
github.com/thetatoken/theta/consensus.(*ConsensusEngine).mainLoop(0xc00ba5e180)
/go/src/github.com/thetatoken/theta/consensus/engine.go:283 +0x6ce
created by github.com/thetatoken/theta/consensus.(*ConsensusEngine).Start
/go/src/github.com/thetatoken/theta/consensus/engine.go:162 +0x1ed
Any clues?
Ty.
Thanks for reporting the issue. This is strange, looks like the DB was corrupted somehow. One option is to restart the node from the latest snapshot
# Replace <CONFIG_FOLDER> with the actual path to your config folder
move <CONFIG_FOLDER>/db <CONFIG_FOLDER>/db.old
move <CONFIG_FOLDER>/snapshot <CONFIG_FOLDER>/snapshot.old
wget -O <CONFIG_FOLDER>/snapshot `curl -k https://mainnet-data.thetatoken.org/snapshot`
Then restart the Theta node as usual. Thanks.
Thank you for your response, I had to delete db folder in bin/ as well. Now I'm repeatedly getting this
[2022-09-15 16:46:32] DEBUG [request] refreshCounter: 3
[2022-09-15 16:46:32] DEBUG [request] Reset refreshCounter
[2022-09-15 16:46:32] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:32] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:32] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:35] DEBUG [consensus] Epoch timeout. Repeating epoch e.epoch=17185574
[2022-09-15 16:46:35] DEBUG [consensus] Enter epoch 17185574
[2022-09-15 16:46:35] DEBUG [consensus] shouldPropose=false: epoch is behind epoch=17185574 tip=0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552 tip.Epoch=17185574
[2022-09-15 16:46:39] DEBUG [request] refreshCounter: 0
[2022-09-15 16:46:39] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:39] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:39] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:45] DEBUG [request] refreshCounter: 1
[2022-09-15 16:46:45] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:45] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:45] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:47] DEBUG [consensus] Epoch timeout. Repeating epoch e.epoch=17185574
[2022-09-15 16:46:47] DEBUG [consensus] Enter epoch 17185574
[2022-09-15 16:46:47] DEBUG [consensus] shouldPropose=false: epoch is behind epoch=17185574 tip=0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552 tip.Epoch=17185574
[2022-09-15 16:46:51] DEBUG [request] refreshCounter: 2
[2022-09-15 16:46:51] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:51] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:51] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:58] DEBUG [request] refreshCounter: 3
[2022-09-15 16:46:58] DEBUG [request] Reset refreshCounter
[2022-09-15 16:46:58] DEBUG [request] Resampled peers to send requests: []
[2022-09-15 16:46:58] DEBUG [request] Sending inventory request channelID=3 end= peers=[] starts=[0x59496790d1653c342b509fcadd0e50c83bc184809919ab231e72ed2aff4c4552]
[2022-09-15 16:46:58] DEBUG [p2p] Broadcasting message to 0 neighbors on channel 3, skipEdgeNode: true
[2022-09-15 16:46:59] INFO [p2p] Connecting to seed and persisted peers...
[2022-09-15 16:46:59] WARN [p2p] Failed to retrieve previously persisted peers
[2022-09-15 16:46:59] DEBUG Memory usage: Alloc = 713.227 MiB TotalAlloc = 148619.333 MiB Sys = 2352.441 MiB NumGC = 312 StackInuse = 1.031 MiB StackSys = 1.031 MiB HeapInuse = 800.391 MiB HeapSys = 2238.969 MiB
Is this normal?
Probably this is my issue?
Could not perform UPNP discover: write udp4 0.0.0.0:57859->239.255.255.250:1900: i/o timeout
I tried from different VPS but I get the same message.
For a really strange reason my config.yaml was missing!!
Downloaded and all good!
You can close it, thank you.