Error in concurrent map accesses

[zuowang]

Description

fatal error: concurrent map read and map write

goroutine 1573695 [running]: runtime.throw(0xf1a960, 0x21) /opt/go/src/runtime/panic.go:530 +0x90 fp=0xc820847860 sp=0xc820847848 runtime.mapaccess1_fast64(0xbb6f60, 0xc820862ae0, 0x4, 0xc82016b5f0) /opt/go/src/runtime/hashmap_fast.go:112 +0x5a fp=0xc820847880 sp=0xc820847860 github.com/hyperledger/fabric/core/ledger/statemgmt/buckettree.(_bucketTreeDelta).getOrCreateBucketNode(0xc8231de3d8, 0xc82079f120, 0x22) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/statemgmt/buckettree/bucket_tree_delta.go:30 +0x4f fp=0xc820847920 sp=0xc820847880 github.com/hyperledger/fabric/core/ledger/statemgmt/buckettree.(_StateImpl).processBucketTreeDelta(0xc8203f8000, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/statemgmt/buckettree/state_impl.go:165 +0x78c fp=0xc820847a38 sp=0xc820847920 github.com/hyperledger/fabric/core/ledger/statemgmt/buckettree.(_StateImpl).ComputeCryptoHash(0xc8203f8000, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/statemgmt/buckettree/state_impl.go:115 +0x114 fp=0xc820847a98 sp=0xc820847a38 github.com/hyperledger/fabric/core/ledger/statemgmt/state.(_State).GetHash(0xc8203f8050, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/statemgmt/state/state.go:237 +0x11d fp=0xc820847b08 sp=0xc820847a98 github.com/hyperledger/fabric/core/ledger.(_Ledger).GetTempStateHash(0xc8203f04a0, 0x0, 0x0, 0x0, 0x0, 0x0) /opt/gopath/src/github.com/hyperledger/fabric/core/ledger/ledger.go:212 +0x42 fp=0xc820847b40 sp=0xc820847b08 github.com/hyperledger/fabric/core/chaincode.ExecuteTransactions(0x7feb422a67c8, 0xc820010578, 0xe40988, 0x7, 0xc8231de248, 0x1, 0x1, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/core/chaincode/exectransaction.go:153 +0x473 fp=0xc820847c60 sp=0xc820847b40 github.com/hyperledger/fabric/consensus/helper.(_Helper).ExecTxs(0xc820400690, 0xb95900, 0xc8204e94c0, 0xc8231de248, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, ...) /opt/gopath/src/github.com/hyperledger/fabric/consensus/helper/helper.go:182 +0xe3 fp=0xc820847e58 sp=0xc820847c60 github.com/hyperledger/fabric/consensus/obcpbft.(_obcBatch).executeImpl(0xc820286e40, 0xc8231de248, 0x1, 0x1, 0xc8232a0d00, 0x4, 0x8) /opt/gopath/src/github.com/hyperledger/fabric/consensus/obcpbft/obc-batch.go:242 +0x1c0 fp=0xc820847f48 sp=0xc820847e58 runtime.goexit() /opt/go/src/runtime/asm_amd64.s:1998 +0x1 fp=0xc820847f50 sp=0xc820847f48 created by github.com/hyperledger/fabric/consensus/obcpbft.(_obcBatch).execute /opt/gopath/src/github.com/hyperledger/fabric/consensus/obcpbft/obc-batch.go:236 +0x88c

Describe How to Reproduce

1. start 4 validator peer in vagrant on single node. docker run --rm -it -e CORE_VM_ENDPOINT=http://172.17.0.1:2375 -e CORE_PEER_ID=vp0 -e CORE_PEER_ADDRESSAUTODETECT=true -e CORE_SECURITY_ENABLED=true -e CORE_SECURITY_PRIVACY=true -e CORE_PEER_PKI_ECA_PADDR=172.17.0.1:50051 -e CORE_PEER_PKI_TCA_PADDR=172.17.0.1:50051 -e CORE_PEER_PKI_TLSCA_PADDR=172.17.0.1:50051 -e CORE_SECURITY_ENROLLID=test_vp0 -e CORE_SECURITY_ENROLLSECRET=MwYpmSRjupbT -e CORE_PEER_VALIDATOR_CONSENSUS_PLUGIN=pbft -e CORE_PBFT_GENERAL_MODE=batch -e CORE_LOGGING_LEVEL=CRITICAL -e CORE_PEER_PROFILE_ENABLED=true -p 30303:30303 -p 31315:31315 hyperledger/fabric-peer peer node start
2. start 50 thread to Invoke chaincode_example02 from command line CORE_PEER_ADDRESS=172.17.0.2:30303 CORE_SECURITY_ENABLED=true CORE_SECURITY_PRIVACY=true build/bin/peer chaincode invoke -u jim -n 3ad1fc0c484709031dc75e9f0fe432a1b4940f6cdabd2484c4dfe457666d58dc93e968d4eb444fff39bf6e47b0baa1d6b4948010d46276af4485e9ea035e3299 -c '{"Function":"invoke", "Args": ["a","b","1"]}'

[srderson]

@manish-sethi could you please investigate

[manish-sethi]

@zuowang I could not reproduce this with the latest source code. The panic stack that you pasted could be caused if state modification is being done concurrently - a related bug was fixed in commit 6739665 (see #1865). Can you try reproducing this with the latest source? If you can reproduce this issue, Please paste the dump of other goroutines as well?