klaytn icon indicating copy to clipboard operation
klaytn copied to clipboard

Consensus error occured while rolling updating serviceChain

Open hqjang-pepper opened this issue 2 years ago • 1 comments

Describe the bug Consensus error occured while rolling updating serviceChain

INFO[08/27,20:45:43 +09] [33] Added a multichannel P2P Peer             id=09fc43f2363a090a conn=inbound    peerID=09fc43f2363a090a
DEBUG[08/27,20:45:44 +09] [33] Fetching a new block header               id=09fc43f2363a090a conn=inbound    hash=c39bfd…d74ced
DEBUG[08/27,20:45:44 +09] [29] Queued propagated block                   peer=09fc43f2363a090a number=5168422 hash=c39bfd…d74ced queued=2
DEBUG[08/27,20:45:45 +09] [33] Fetching a new block header               id=09fc43f2363a090a conn=inbound    hash=4f9715…699c74
DEBUG[08/27,20:45:45 +09] [29] Queued propagated block                   peer=09fc43f2363a090a number=5168423 hash=4f9715…699c74 queued=3
DEBUG[08/27,20:45:46 +09] [33] Fetching a new block header               id=09fc43f2363a090a conn=inbound    hash=e9f75d…2a41fd
DEBUG[08/27,20:45:46 +09] [29] Queued propagated block                   peer=09fc43f2363a090a number=5168424 hash=e9f75d…2a41fd queued=4
DEBUG[08/27,20:45:47 +09] [33] Fetching a new block header               id=09fc43f2363a090a conn=inbound    hash=2e1d7f…52f521
DEBUG[08/27,20:45:47 +09] [29] Queued propagated block                   peer=09fc43f2363a090a number=5168425 hash=2e1d7f…52f521 queued=5
DEBUG[08/27,20:45:48 +09] [33] Fetching a new block header               id=d03dd880acf16744 conn=inbound    hash=a6b4ac…6cf14e
DEBUG[08/27,20:45:48 +09] [29] Queued propagated block                   peer=d03dd880acf16744 number=5168426 hash=a6b4ac…6cf14e queued=6
INFO[08/27,20:45:48 +09] [28] Block synchronisation started
DEBUG[08/27,20:45:48 +09] [28] Synchronising with the network            peer=d03dd880acf16744 klay=65 head=a6b4ac…6cf14e td=5168426 mode=full
INFO[08/27,20:45:48 +09] [51] Mining aborted due to sync
DEBUG[08/27,20:45:48 +09] [28] Retrieving remote chain height            peer=d03dd880acf16744
DEBUG[08/27,20:45:48 +09] [33] Fetching batch of headers                 id=d03dd880acf16744 conn=inbound    count=1 fromhash=a6b4ac…6cf14e skip=63 reverse=true
DEBUG[08/27,20:45:48 +09] [28] Remote head identified, no pivot          peer=d03dd880acf16744 number=5168426 hash=a6b4ac…6cf14e
DEBUG[08/27,20:45:48 +09] [28] Looking for common ancestor               peer=d03dd880acf16744 local=5168268 remote=5168426
DEBUG[08/27,20:45:48 +09] [33] Fetching batch of headers                 id=d03dd880acf16744 conn=inbound    count=13 fromnum=5168076 skip=15 reverse=false
DEBUG[08/27,20:45:48 +09] [28] Found common ancestor                     peer=d03dd880acf16744 number=5168268 hash=7efc2c…c32fb6
DEBUG[08/27,20:45:48 +09] [28] spawnSync started                         peerID=d03dd880acf16744
DEBUG[08/27,20:45:48 +09] [28] Processing full sync content
DEBUG[08/27,20:45:48 +09] [28] Downloading transaction receipts          origin=5168269
DEBUG[08/27,20:45:48 +09] [28] Downloading block bodies                  origin=5168269
DEBUG[08/27,20:45:48 +09] [28] Directing header downloads                peer=d03dd880acf16744 origin=5168269
DEBUG[08/27,20:45:48 +09] [28] Processing headers                        origin=5168269 td=5168426
DEBUG[08/27,20:45:48 +09] [28] Downloading staking information           origin=5168269
DEBUG[08/27,20:45:48 +09] [33] Fetching batch of headers                 id=d03dd880acf16744 conn=inbound    count=128 fromnum=5168460 skip=191 reverse=false
DEBUG[08/27,20:45:48 +09] [33] Fetching batch of headers                 id=d03dd880acf16744 conn=inbound    count=192 fromnum=5168427 skip=0   reverse=false
DEBUG[08/27,20:45:48 +09] [28] No more headers available                 peer=d03dd880acf16744
DEBUG[08/27,20:45:48 +09] [28] Header download terminated                peer=d03dd880acf16744 elapsed=2.941903ms
DEBUG[08/27,20:45:48 +09] [28] Processing headers terminated             origin=5168427 td=5168426 elapsed=4.073587ms
INFO[08/27,20:45:48 +09] [28] Downloader queue stats                    receiptTasks=0 blockTasks=106 stakingInfoTasks=0 itemSize=889.15B throttle=8192
DEBUG[08/27,20:45:48 +09] [28] Inserting downloaded chain                items=39 firstnum=5168269 firsthash=567f70…2bbd9d lastnum=5168307 lasthash=951d90…e7e10e
DEBUG[08/27,20:45:48 +09] [33] Fetching batch of block bodies            id=09fc43f2363a090a conn=inbound    count=2
DEBUG[08/27,20:45:48 +09] [33] Fetching batch of block bodies            id=d03dd880acf16744 conn=inbound    count=2
DEBUG[08/27,20:45:48 +09] [28] Data fetching completed                   type=receipts
DEBUG[08/27,20:45:48 +09] [28] Transaction receipt download terminated   err=nil                                                    elapsed=5.047801ms
DEBUG[08/27,20:45:48 +09] [28] Data fetching completed                   type=stakingInfos
DEBUG[08/27,20:45:48 +09] [28] Staking information download terminated   err=nil                                                    elapsed=4.948659ms
DEBUG[08/27,20:45:48 +09] [28] Processing full sync content terminated   elapsed=5.878151ms
panic: sync/atomic: store of nil value into Value

goroutine 1294 [running]:
sync/atomic.(*Value).Store(0xc013821d08, 0x0, 0x0)
        /usr/local/go/src/sync/atomic/value.go:47 +0x12c
github.com/klaytn/klaytn/consensus/istanbul/validator.(*defaultSet).Copy(0xc010a6dc80, 0x0, 0x0)
        /ext-go/1/src/github.com/klaytn/klaytn/consensus/istanbul/validator/default.go:362 +0x259
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).verifyCommittedSeals(0xc000691800, 0x1fff560, 0xc000214000, 0xc013858240, 0xc002c7ab10, 0x1, 0x1, 0x0, 0x0)
        /ext-go/1/src/github.com/klaytn/klaytn/consensus/istanbul/backend/engine.go:354 +0x16a
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).verifyCascadingFields(0xc000691800, 0x1fff560, 0xc000214000, 0xc013858240, 0xc002c7ab10, 0x1, 0x1, 0xc0111bc010, 0x4632c5)
        /ext-go/1/src/github.com/klaytn/klaytn/consensus/istanbul/backend/engine.go:280 +0x316
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).verifyHeader(0xc000691800, 0x1fff560, 0xc000214000, 0xc013858240, 0xc002c7ab10, 0x1, 0x1, 0x1, 0xc010f77b40)
        /ext-go/1/src/github.com/klaytn/klaytn/consensus/istanbul/backend/engine.go:245 +0x5ef
github.com/klaytn/klaytn/consensus/istanbul/backend.(*backend).VerifyHeader(0xc000691800, 0x1fff560, 0xc000214000, 0xc013858240, 0x9dbd2b43bf6a6901, 0xc0001e5650, 0xc0111ae9a8)
        /ext-go/1/src/github.com/klaytn/klaytn/consensus/istanbul/backend/engine.go:182 +0x145
github.com/klaytn/klaytn/blockchain.(*BlockChain).insertChain(0xc000214000, 0xc0110a03c0, 0x27, 0x27, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
        /ext-go/1/src/github.com/klaytn/klaytn/blockchain/blockchain.go:1887 +0x54aa
github.com/klaytn/klaytn/blockchain.(*BlockChain).InsertChain(0xc000214000, 0xc0110a03c0, 0x27, 0x27, 0x0, 0x0, 0xa)
        /ext-go/1/src/github.com/klaytn/klaytn/blockchain/blockchain.go:1752 +0x67
github.com/klaytn/klaytn/datasync/downloader.(*Downloader).importBlockResults(0xc00c6fc1e0, 0xc0110a0280, 0x27, 0x27, 0x27, 0x0)
        /ext-go/1/src/github.com/klaytn/klaytn/datasync/downloader/downloader.go:1586 +0x539
github.com/klaytn/klaytn/datasync/downloader.(*Downloader).processFullSyncContent(0xc00c6fc1e0, 0x0, 0x0)
        /ext-go/1/src/github.com/klaytn/klaytn/datasync/downloader/downloader.go:1560 +0x10a
github.com/klaytn/klaytn/datasync/downloader.(*Downloader).spawnSync.func1(0xc00c6fc1e0, 0xc0111801e0, 0xc01010aa20)
        /ext-go/1/src/github.com/klaytn/klaytn/datasync/downloader/downloader.go:556 +0x5b
created by github.com/klaytn/klaytn/datasync/downloader.(*Downloader).spawnSync
        /ext-go/1/src/github.com/klaytn/klaytn/datasync/downloader/downloader.go:556 +0x16f

How to reproduce 4SCN + single mode operating removevalidator, addvalidator one by one In the second node, after the binary update, execute kscnd start and the process ends after the panic log is generated.

Expected behavior binary start with no error

Attachments If applicable, add data, logs or screenshots to help explain your problem.

Environment (please complete the following information)

  • previous version -> 1.9.1

Additional context Add any other context about the problem here.

hqjang-pepper avatar Aug 29 '22 07:08 hqjang-pepper

It seems that this issues is the same with #1206

aidan-kwon avatar Aug 29 '22 08:08 aidan-kwon

Panic is disappeared with #1600

yoomee1313 avatar Nov 15 '22 01:11 yoomee1313