fabric icon indicating copy to clipboard operation
fabric copied to clipboard
verified publisher icon hyperledger

Creating a channel with the same name again causes Orderer process to terminate abnormally

Open takayuki-nagai opened this issue 9 months ago • 5 comments

Description

After deleting a channel from all Orderers and Peers, creating a channel with the same name again causes Orderer process to terminate abnormally.

Steps to reproduce

Environment: Fabric v2.5.9, 3 Raft Orderers, 1 PeerOrg with 2 Peers (no System Channel)

  1. Create an application channel.
  2. Use "osnadmin channel remove" and "peer node unjoin" commands to remove the Channel from the Orderer and Peer.
  3. After the information about that Channel has disappeared from the network, create a channel with the same name again.

In step 3, I created a new genesis block using the "configtxgen" command and tried to create a channel. However, the following hash value inconsistency error occurred at Orderer, and the process went down. panic: [channel: <Channel name>] Could not append block: unexpected Previous block hash. Expected PreviousHash = [<Hash value A>], PreviousHash referred in the latest block= [<Hash value B>] "Hyerledger Fabric Docs" does not state whether it is okay to create a channel with the same name as a previously created channel, but I think that the Orderer process should be prevented from abnormal termination.

Related issue

Issue #4016 reports that WAL & snapshots data remains on Raft Orderer after a channel is deleted. This bug has not been fixed at this time. When I deleted old WAL & snapshots directories on Raft Orderer in above step 2 and created the channel again, the channel was created successfully. Therefore, I assume that Orderer process down is related to the problem reported in this issue. I would appreciate if anyone could investigate whether this assumption is correct.

Full text of Orderer's crush log

2025-03-26 14:43:42.536 JST [orderer.consensus.etcdraft] writeBlock -> INFO 107 Writing block [1] (Raft index: 5) to ledger channel=forstaging2029 node=2
2025-03-26 14:43:42.543 JST [orderer.commmon.multichannel] commitBlock -> PANI 108 [channel: forstaging2029] Could not append block: unexpected Previous block hash. Expected PreviousHash = [a65f4ca5f9be4c1aefa72768399af0eba9177507ca3f77d6a71822d35f70ce25], PreviousHash referred in the latest block= [e47209331224f357b8c98446fa4a36f9d56d7b5456cb52f52d69d6e255cca2d9]
panic: [channel: forstaging2029] Could not append block: unexpected Previous block hash. Expected PreviousHash = [a65f4ca5f9be4c1aefa72768399af0eba9177507ca3f77d6a71822d35f70ce25], PreviousHash referred in the latest block= [e47209331224f357b8c98446fa4a36f9d56d7b5456cb52f52d69d6e255cca2d9]
goroutine 179 [running]:
go.uber.org/zap/zapcore.CheckWriteAction.OnWrite(0x0?, 0x0?, {0x0?, 0x0?, 0xc007bc7d00?})
#011/vendor/go.uber.org/zap/zapcore/entry.go:196 +0x54
go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc007aa2dd0, {0x0, 0x0, 0x0})
#011/vendor/go.uber.org/zap/zapcore/entry.go:262 +0x24e
go.uber.org/zap.(*SugaredLogger).log(0xc000115ce8, 0x4, {0x11f0e60?, 0x0?}, {0xc007cf9598?, 0x1?, 0xdae96f?}, {0x0, 0x0, 0x0})
#011/vendor/go.uber.org/zap/sugar.go:355 +0xec
go.uber.org/zap.(*SugaredLogger).Panicf(...)
#011/vendor/go.uber.org/zap/sugar.go:229
github.com/hyperledger/fabric/common/flogging.(*FabricLogger).Panicf(...)
#011/common/flogging/zap.go:74
github.com/hyperledger/fabric/orderer/common/multichannel.(*BlockWriter).commitBlock(0xc007884000, {0xc007c99736, 0x9, 0x9})
#011/orderer/common/multichannel/blockwriter.go:204 +0x14f
github.com/hyperledger/fabric/orderer/common/multichannel.(*BlockWriter).WriteBlockSync(0xc007d21500?, 0xc007c99740?, {0xc007c99736?, 0x13a3b78?, 0xc007cd9020?})
#011/orderer/common/multichannel/blockwriter.go:193 +0xe5
github.com/hyperledger/fabric/orderer/common/multichannel.(*BlockWriter).WriteConfigBlock(0xc007884000, 0xc007bccd40, {0xc007c99736, 0x9, 0x9})
#011/orderer/common/multichannel/blockwriter.go:159 +0x81d
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).writeConfigBlock(0xc0078f0008, 0xc007bccd40, 0x5)
#011/orderer/consensus/etcdraft/chain.go:1317 +0x2fa
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).writeBlock(0xc0078f0008, 0xc007bccd40, 0x5)
#011/orderer/consensus/etcdraft/chain.go:897 +0x2e9
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).apply(0xc0078f0008, {0xc0078dc038, 0xa, 0x0?})
#011/orderer/consensus/etcdraft/chain.go:1142 +0x2b3
github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).run(0xc0078f0008)
#011/orderer/consensus/etcdraft/chain.go:805 +0xc13
created by github.com/hyperledger/fabric/orderer/consensus/etcdraft.(*Chain).Start in goroutine 1
#011/orderer/consensus/etcdraft/chain.go:376 +0x289

takayuki-nagai avatar Apr 04 '25 05:04 takayuki-nagai

I want to try to replicate your situation and fix it. I have a few questions:

  • have you tried this in fabric-samples
  • tried it in the latest version (v3.1.0)
  • have you tried to do it with bft consensus

pfi79 avatar Apr 10 '25 07:04 pfi79

@pfi79 Thanks for your comments.

have you tried this in fabric-samples

No, the above experiments were performed on my original environment, not on fabric-samples environment.

tried it in the latest version (v3.1.0) have you tried to do it with bft consensus

I had not experimented with v3.x or bft orderer, so I tried it. However, even if I deleted the data in the following two folders, I was not able to recreate the channel. It seems that some data about old channel remains in other files or folders.

  • /var/hyperledger/production/orderer/etcdraft/wal
  • /var/hyperledger/production/orderer/etcdraft/snapshot

takayuki-nagai avatar Apr 14 '25 10:04 takayuki-nagai

@takayuki-nagai let's refine the scenario:

  • created a genesis block
  • create a channel on the orderers
  • create a channel on peers.

Did you do any other actions?

  • deleted the channel on the peers
  • deleted the channel on the orderers
  • deleted the necessary folders ......./orderer/etcdraft/wal/name_channel ......./orderer/etcdraft/snapshot/name_channel

That is, after this, there is no channel left on any node. It's as if it never existed.

Did you do anything else?

  • created a genesis block
  • create a channel on the orderers - an error occurs here

right?

pfi79 avatar Apr 14 '25 17:04 pfi79

@pfi79 Thanks for your comment.

Sorry, but unlike my previous comment, I could compretely delete the old channel as you say and recreate the channel with same name by following the steps below. I think I missed something when I tried it before. Here are the details of the steps I tried:

  1. Download latest Hyperledger Fabric (v3.1.0) and fabric-samples, create bft orderers and peers, create "mychannel", and deploy "basic" chainode. cd fabric-samples/test-network ./network.sh up createChannel -ca -c mychannel -bft ./network.sh deployCC -ccn basic -ccp ../asset-transfer-basic/chaincode-javascript/ -ccl javascript

  2. Delete the channel on the orderers. osnadmin channel remove -c mychannel -o localhost:7053 --ca-file /home/ubuntu/fabric-samples/test-network/organizations/ordererOrganizations/example.com/tlsca/tlsca.example.com-cert.pem --client-cert /home/ubuntu/fabric-samples/test-network/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.crt --client-key /home/ubuntu/fabric-samples/test-network/organizations/ordererOrganizations/example.com/orderers/orderer.example.com/tls/server.key Repeat this command for all four orderers.

  3. Delete the channel on the peers. vi compose/compose-bft-test-net.yaml Edit the startup commands for the two peer containers are as follows: "peer node start" -> "peer unjoin -c mychannel" Restart the peers. docker-compose -f compose/compose-bft-test-net.yaml -f compose/docker/docker-compose-bft-test-net.yaml up -d Change the startup command back to its original state are as follows: "peer unjoin -c mychannel" -> "peer node start" Restart the peer containers. docker-compose -f compose/compose-bft-test-net.yaml -f compose/docker/docker-compose-bft-test-net.yaml up -d

  4. Delete "mychannel" folders remaining on the orderers. docker exec -it orderer.example.com bash rm -rf /var/hyperledger/production/orderer/etcdraft/wal/mychannel (When I tried this procedure, there were no files in "......./orderer/etcdraft/snapshot/" folder.) Repeat this command for all four orderers, and restart orderer containers. docker restart orderer.example.com orderer2.example.com orderer3.example.com orderer4.example.com

  5. Delete the old genesis block and then recreate the channel. rm -rf channel-artifacts ./network.sh createChannel -c mychannel ./network.sh deployCC -ccn basic -ccp ../asset-transfer-basic/chaincode-javascript/ -ccl javascript

In conclusion, I think it would be desirable to modify the latest Hyperledger Fabric so that the specific folder deletion procedure shown in step 4 is completed during step 2.

takayuki-nagai avatar Apr 15 '25 06:04 takayuki-nagai

I want to write a test and run it in brunch main, tag v3.1.0 and brunch release-2.5, tag v2.5.12. And check if it works everywhere or not.

pfi79 avatar Apr 15 '25 17:04 pfi79