Panic in 3.0.0

[acroca]

A node running 3.0.0 got a panic while it was running normally:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x86fb9f]

goroutine 7542 [running]:
github.com/kowala-tech/kcoin/client/trie.(*Database).insert(0xc420502780, 0xffd3f51a5d1de87e, 0xf2f7902c430cd693, 0xf9f700e9afecf541, 0x7ccbb39d620b0268, 0xc43781a000, 0x74, 0x226, 0x1b72fa0, 0xc4376c22d0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/database.go:300 +0x3cf
github.com/kowala-tech/kcoin/client/trie.(*hasher).store(0xc42258e640, 0x1b72fa0, 0xc4376c22d0, 0xc420502780, 0x1b72f00, 0xc4376c22d0, 0x1b72fa0, 0xc4376c2320, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:174 +0x2e7
github.com/kowala-tech/kcoin/client/trie.(*hasher).hash(0xc42258e640, 0x1b72fa0, 0xc4376c2230, 0xc420502780, 0xc43b0c5e00, 0x7ae9c3, 0xc43b0c5e38, 0x4d93ef, 0x10de660, 0x10de660, ...)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:82 +0x17a
github.com/kowala-tech/kcoin/client/trie.(*hasher).hashChildren(0xc42258e640, 0x1b72f20, 0xc437822a00, 0xc420502780, 0x1b75d01, 0x1b75d20, 0xc437734ac0, 0x0, 0x0, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:132 +0x18f
github.com/kowala-tech/kcoin/client/trie.(*hasher).hash(0xc42258e640, 0x1b72f20, 0xc437822a00, 0xc420502780, 0x0, 0x1b75d20, 0xc437734ac0, 0x1b75d20, 0xc437734ac0, 0x0, ...)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:78 +0x10f
github.com/kowala-tech/kcoin/client/trie.(*hasher).hashChildren(0xc42258e640, 0x1b72f20, 0xc437822780, 0xc420502780, 0x1b75d01, 0x1b75d20, 0xc437734640, 0x0, 0x0, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:132 +0x18f
github.com/kowala-tech/kcoin/client/trie.(*hasher).hash(0xc42258e640, 0x1b72f20, 0xc437822780, 0xc420502780, 0xc42053ca00, 0x1b75d20, 0xc437734640, 0x1b75d20, 0xc437734640, 0x0, ...)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:78 +0x10f
github.com/kowala-tech/kcoin/client/trie.(*hasher).hashChildren(0xc42258e640, 0x1b72f20, 0xc437822500, 0xc420502780, 0xc435433301, 0xc43b0c6330, 0xc43b0c64c8, 0x773b20, 0xc420554060, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:132 +0x18f
github.com/kowala-tech/kcoin/client/trie.(*hasher).hash(0xc42258e640, 0x1b72f20, 0xc437822500, 0xc420502780, 0xc436eedf01, 0x30, 0x30, 0x108aec0, 0x18, 0x800378204c0, ...)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/hasher.go:78 +0x10f
github.com/kowala-tech/kcoin/client/trie.(*Trie).hashRoot(0xc4361e3b20, 0xc420502780, 0xc4219a6210, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/trie.go:459 +0x113
github.com/kowala-tech/kcoin/client/trie.(*Trie).Commit(0xc4361e3b20, 0xc4219a6210, 0x0, 0x0, 0x0, 0x0, 0x0, 0x100000000425567)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/trie.go:444 +0x8e
github.com/kowala-tech/kcoin/client/trie.(*SecureTrie).Commit(0xc4361e3b20, 0xc4219a6210, 0x0, 0x0, 0x0, 0x0, 0x20, 0x20)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/trie/secure_trie.go:139 +0x549
github.com/kowala-tech/kcoin/client/core/state.cachedTrie.Commit(0xc4361e3b20, 0xc420513620, 0xc4219a6210, 0x0, 0x0, 0x0, 0x0, 0x0, 0xc43545c158)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/core/state/database.go:153 +0x7d
github.com/kowala-tech/kcoin/client/core/state.(*cachedTrie).Commit(0xc4219a61a0, 0xc4219a6210, 0x0, 0x0, 0x0, 0x0, 0x3b9662a98ec86d68, 0xf56a1edbce51299a)
	<autogenerated>:1 +0x9f
github.com/kowala-tech/kcoin/client/core/state.(*StateDB).Commit(0xc43affa380, 0xc420076401, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/core/state/statedb.go:612 +0x5f7
github.com/kowala-tech/kcoin/client/core.(*BlockChain).WriteBlockWithState(0xc42016e000, 0xc43551f110, 0x0, 0x0, 0x0, 0xc43affa380, 0x0, 0x0, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/core/blockchain.go:844 +0x27a
github.com/kowala-tech/kcoin/client/core.(*BlockChain).insertChain(0xc42016e000, 0xc43b842000, 0x7fe, 0x7fe, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/core/blockchain.go:1101 +0x1e6a
github.com/kowala-tech/kcoin/client/core.(*BlockChain).InsertChain(0xc42016e000, 0xc43b842000, 0x7fe, 0x7fe, 0xc436300900, 0xc421a58d20, 0x7a0128)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/core/blockchain.go:956 +0x67
github.com/kowala-tech/kcoin/client/knode/downloader.(*Downloader).importBlockResults(0xc420064680, 0xc437e24000, 0x7fe, 0x7fe, 0x7fe, 0x0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/knode/downloader/downloader.go:1342 +0x98a
github.com/kowala-tech/kcoin/client/knode/downloader.(*Downloader).processFullSyncContent(0xc420064680, 0xc4204dbf98, 0xc420060060)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/knode/downloader/downloader.go:1316 +0x79
github.com/kowala-tech/kcoin/client/knode/downloader.(*Downloader).(github.com/kowala-tech/kcoin/client/knode/downloader.processFullSyncContent)-fm(0x8, 0x11cf0e0)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/knode/downloader/downloader.go:450 +0x2a
github.com/kowala-tech/kcoin/client/knode/downloader.(*Downloader).spawnSync.func1(0xc420064680, 0xc42253a660, 0xc421bb6000)
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/knode/downloader/downloader.go:462 +0x59
created by github.com/kowala-tech/kcoin/client/knode/downloader.(*Downloader).spawnSync
	/go/src/github.com/kowala-tech/kcoin/client/build/_workspace/src/github.com/kowala-tech/kcoin/client/knode/downloader/downloader.go:462 +0xb6

[acroca]

/cc @JekaMas @rgeraldes

[acroca]

I think I have more details.

There were some nodes running 2.0.1 with some new blocks (like 200 blocks). There was one node with old blocks (like 150k blocks) that I forgot to stop.

All nodes were in sync with the first ones (200 blocks) but at some point some node found the one I forgot to stop and this panic happened.

[JekaMas]

Hmmm... @acroca it sounds like a good e2e test to have! Shall we add one?

[acroca]

if the reason is related with the node with more blocks, I wouldn't bother. It's quite unlikely to happen on an ongoing network

[JekaMas]

but someone could do it as an attack on the network.

[acroca]

I think this is something we should cover in unit tests first. But you're right we could have an e2e test that simulates some kind of fork and puts some expectations in place to make sure it all behaves as expected. Although I think fixing this kind of panics is something that should go to unit tests instead.

[JekaMas]

Agree! Lets close this issue and make a new one for unit and e2e tests.

[acroca]

This issue can be closed once we have a PR implementing the fix and a unit test for it, don't you think?