fabric icon indicating copy to clipboard operation
fabric copied to clipboard

orderer crashes down after receiving fuzzed channel tx.

Open SecTechTool opened this issue 3 years ago • 3 comments

System information: version: fabric 2.3 OS: ubuntu 20.04

Experiments setup: Using first-network in https://github.com/SmartBFT-Go/fabric-samples. Repeatedly sending fuzzed channel tx with the same Channel name.

Expected behavior: Orderers run as usual.

Actual behaviour: orderers crash down, and can not be recovered.

Bug location: https://github.com/SmartBFT-Go/fabric/blob/release-2.3-bft/orderer/common/multichannel/registrar.go

// CreateChain makes the Registrar create a consensus.Chain with the given name.
func (r *Registrar) CreateChain(chainName string) {
	lf, err := r.ledgerFactory.GetOrCreate(chainName)
	if err != nil {
		logger.Panicf("Failed obtaining ledger factory for %s: %v", chainName, err)
	}
	chain := r.GetChain(chainName)
	if chain != nil {
		logger.Infof("A chain of type %T for channel %s already exists. "+
			"Halting it.", chain.Chain, chainName)
		chain.Halt()
	}
	r.newChain(configTx(lf))
}

In line 529, should not panic directly.

log info:

2022-11-06 03:12:09.321 UTC [orderer.commmon.multichannel] createNewChain -> PANI 028 Error creating chain support: error creating consenter for channel: fuzztest: failed to restore persisted raft data: failed to create or read WAL: failed to open WAL: fileutil: file already locked
panic: Error creating chain support: error creating consenter for channel: fuzztest:  failed to create or read WAL: failed to open WAL: fileutil: file already locked

goroutine 118 [running]:
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00002c000, 0x0, 0x0, 0x0)
/go/src/github.com/hyperledger/fabric/vendor/go.uber.org/zap/zapcore/entry.go:230 +0x545

SecTechTool avatar Nov 07 '22 02:11 SecTechTool

I remember that to create a channel you need to sign the transaction by enough administrators in the network. I think you can restrict it to a threshold of a majority, so I am not sure this is possible in a production network.

In any case, can you check the same problem on the official Fabric with Raft?

yacovm avatar Nov 09 '22 10:11 yacovm

Sorry for the delayed response.

In the oficial Fabric with Raft, there are a locking mechanism in function 'CreateChain' and a exsiting name check in function 'newChain' to prevent this problem. The detailed code is in : https://github.com/hyperledger/fabric/commit/9da0410f622e49d92750ace50246ac75c381ac0a#

func (r *Registrar) CreateChain(chainName string) {
	lf, err := r.ledgerFactory.GetOrCreate(chainName)
	if err != nil {
		logger.Panicf("Failed obtaining ledger factory for %s: %v", chainName, err)
	}
	chain := r.GetChain(chainName)
	if chain != nil {
		logger.Infof("A chain of type %T for channel %s already exists. "+
			"Halting it.", chain.Chain, chainName)
		r.lock.Lock()
		chain.Halt()
		delete(r.chains, chainName)
		r.lock.Unlock()
	}
	r.newChain(configTx(lf))
}
func (r *Registrar) newChain(configtx *cb.Envelope) {
	r.lock.Lock()
	defer r.lock.Unlock()

	channelName, err := channelNameFromConfigTx(configtx)
	if err != nil {
		logger.Warnf("Failed extracting channel name: %v", err)
		return
	}

	// fixes https://github.com/hyperledger/fabric/issues/2931
	if existingChain, exists := r.chains[channelName]; exists {
		if _, isRaftChain := existingChain.Chain.(*etcdraft.Chain); isRaftChain {
			logger.Infof("Channel %s already created, skipping its creation", channelName)
			return
		}
	}

	cs := r.createNewChain(configtx)
	cs.start()
	logger.Infof("Created and started new channel %s", cs.ChannelID())
}

Maybe similar mechnism should be added here?

SecTechTool avatar Nov 11 '22 03:11 SecTechTool

That's why I asked you to check in Fabric... I know this bug.

https://github.com/hyperledger/fabric/pull/2934

Do you want to push a fix?

yacovm avatar Nov 11 '22 20:11 yacovm