fabric icon indicating copy to clipboard operation
fabric copied to clipboard

Panic on leveldb

Open dviejokfs opened this issue 1 year ago • 1 comments

Description

Sometimes, when load is heavy on the peers, the peer crashes with the following error:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x1c pc=0x12df6f8]

goroutine 9216936 [running]:
[github.com/syndtr/goleveldb/leveldb.(*dbIter).Release(0xc00c41dd40)](https://github.com/syndtr/goleveldb/leveldb.(*dbIter).Release(0xc00c41dd40))
 /vendor/[github.com/syndtr/goleveldb/leveldb/db_iter.go:352](https://github.com/syndtr/goleveldb/leveldb/db_iter.go:352) +0x118
[github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*kvScanner).Close(0x0?)](https://github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/statedb/stateleveldb.(*kvScanner).Close(0x0?))
 /core/ledger/kvledger/txmgmt/statedb/stateleveldb/stateleveldb.go:338 +0x1c
[github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*resultsItr).Close(...)](https://github.com/hyperledger/fabric/core/ledger/kvledger/txmgmt/txmgr.(*resultsItr).Close(...))
 /core/ledger/kvledger/txmgmt/txmgr/query_executor.go:495

Steps to reproduce

It's hard to reproduce it since it happens with a lot of load, 300tps/second with chaincode that has custom logic and most likely needs optimization.

But the conditions are:

  • High load on the peers
  • State db (leveldb) under stress
  • Chaincode as a service in Kubernetes

dviejokfs avatar Oct 01 '24 21:10 dviejokfs

The conditions and panic look similar to the ones we're seeing here: https://github.com/hyperledger/fabric/issues/5048

sam-at-luther avatar Nov 01 '24 16:11 sam-at-luther