fablo
fablo copied to clipboard
Published
20 hours ago •
hyperledger-labs
hyperledger-labs
Endorser client failed/refused to connect when joining a channel (remote development)
Greetings!
I have a multi-org single peer network running on a remote single Azure CentOS VM.
SOMETIMES, whenever I try to recreate/reset the network, some peers fail to join the defined channel on fablo-config.json (in logs: stvgd-channel) . This happens occasionally and randomly as the peer that fails to join can be from any of the orgs besides the one creating it (in logs: Inovafil).
E.g., in the logs below, the peer in ASampaio Org fails to join, but sometimes it joins successfully. If that happens however, one of the peers in the remaining organizations can also fail. Listed below are the aforementioned logs:
============ 😻 Creating 'stvgd-channel' on Inovafil/peer0 😻 ==============
Creating channel with name (TLS): stvgd-channel
Orderer: orderer0.ordering-service.inovafil.stvgd.com:7030
CORE_PEER_LOCALMSPID: InovafilMSP
CORE_PEER_ADDRESS: peer0.inovafil.stvgd.com:7021
CORE_PEER_MSPCONFIGPATH: /var/hyperledger/cli/crypto/users/[email protected]/msp
TLS_CA_CERT_PATH is: /var/hyperledger/cli/crypto-orderer/tlsca.inovafil.stvgd.com-cert.pem
CORE_PEER_TLS_CERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.crt
CORE_PEER_TLS_KEY_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.key
CORE_PEER_TLS_ROOTCERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/ca.crt
2023-02-13 16:12:41.807 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2023-02-13 16:12:41.862 UTC [cli.common] readBlock -> INFO 002 Expect block, but got status: &{NOT_FOUND}
2023-02-13 16:12:41.866 UTC [channelCmd] InitCmdFactory -> INFO 003 Endorser and orderer connections initialized
2023-02-13 16:12:42.068 UTC [cli.common] readBlock -> INFO 004 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.072 UTC [channelCmd] InitCmdFactory -> INFO 005 Endorser and orderer connections initialized
2023-02-13 16:12:42.273 UTC [cli.common] readBlock -> INFO 006 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.277 UTC [channelCmd] InitCmdFactory -> INFO 007 Endorser and orderer connections initialized
2023-02-13 16:12:42.478 UTC [cli.common] readBlock -> INFO 008 Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.483 UTC [channelCmd] InitCmdFactory -> INFO 009 Endorser and orderer connections initialized
2023-02-13 16:12:42.686 UTC [cli.common] readBlock -> INFO 00a Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.693 UTC [channelCmd] InitCmdFactory -> INFO 00b Endorser and orderer connections initialized
2023-02-13 16:12:42.895 UTC [cli.common] readBlock -> INFO 00c Expect block, but got status: &{SERVICE_UNAVAILABLE}
2023-02-13 16:12:42.900 UTC [channelCmd] InitCmdFactory -> INFO 00d Endorser and orderer connections initialized
2023-02-13 16:12:43.104 UTC [cli.common] readBlock -> INFO 00e Received block: 0
2023-02-13 16:12:43.177 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2023-02-13 16:12:43.438 UTC [channelCmd] executeJoin -> INFO 002 Successfully submitted proposal to join channel
==== 😸 Joining 'stvgd-channel' on ASampaio/peer0 😸 ====
Fetching channel with name (TLS): stvgd-channel
Orderer: orderer0.ordering-service.inovafil.stvgd.com:7030
CORE_PEER_LOCALMSPID: ASampaioMSP
CORE_PEER_ADDRESS: peer0.asampaio.stvgd.com:7041
CORE_PEER_MSPCONFIGPATH: /var/hyperledger/cli/crypto/users/[email protected]/msp
TLS_CA_CERT_PATH is: /var/hyperledger/cli/crypto-orderer/tlsca.inovafil.stvgd.com-cert.pem
CORE_PEER_TLS_CERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.crt
CORE_PEER_TLS_KEY_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/client.key
CORE_PEER_TLS_ROOTCERT_FILE: /var/hyperledger/cli/crypto/users/[email protected]/tls/ca.crt
2023-02-13 16:12:43.617 UTC [channelCmd] InitCmdFactory -> INFO 001 Endorser and orderer connections initialized
2023-02-13 16:12:43.621 UTC [cli.common] readBlock -> INFO 002 Received block: 0
Error: error getting endorser client for channel: endorser client failed to connect to peer0.asampaio.stvgd.com:7041: failed to create new connection: connection error: desc = "transport: error while dialing: dial tcp 172.21.0.6:7041: connect: connection refused"
My current workaround this issue is to simple iterate through a fablo reset/prune/recreate cycle until it starts working because this issue happens randomly. One thing that I've found is that if I run this locally it always starts successfully with no critical errors.
Any clues on what this might be? I'm thinking it has to do with initializing the docker containers and its timeouts inside the VM but I've decided to consult the creators and the Fablo community, hence this issue.
Cheers and thanks in advance.
We observed similar issues when there was not enough memory available for Docker. What are the limits on your remote VM? (both VM RAM and Docker RAM limits). I feel for a multiorg setup (2-4 orgs, single peer per org) it is save to have like 6-8 GB RAM. (But we don't have any benchmarks for that).
In the current scenario, there are 5 single peer Orgs.
I'm using an 8GB RAM remote VM and each Docker container is limited to 8GB of RAM as well.
For reference, here is the output of docker stats command:
CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS
fd97b05127ee dev-peer0.tintex.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2 0.00% 9.93MiB / 2GiB 0.48% 1.7MB / 1.87MB 0B / 0B 6
ace2ad35f5ba dev-peer0.inovafil.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2 0.00% 10.13MiB / 2GiB 0.49% 1.81MB / 1.97MB 0B / 0B 6
ee2664303a58 dev-peer0.asampaio.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2 0.00% 10.17MiB / 2GiB 0.50% 1.74MB / 1.91MB 0B / 0B 6
b528abfe2489 dev-peer0.domingosdesousa.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2 0.00% 9.977MiB / 2GiB 0.49% 1.79MB / 1.95MB 0B / 0B 6
026ac65fd7e8 dev-peer0.tmg.stvgd.com-stvgd-chaincode_14-9c3b47360606c26eaca4cb5871d6012302098ef77c9c404b190ece492fa2d8d2 0.00% 10.07MiB / 2GiB 0.49% 1.79MB / 1.96MB 0B / 0B 6
f51aeb5af6f7 explorer.example.com 12.41% 436.5MiB / 7.638GiB 5.58% 3.76GB / 1.21GB 2.2MB / 12.3kB 79
9b67acfc9f58 db.explorer.example.com 2.19% 44MiB / 7.638GiB 0.56% 755MB / 368MB 4.1kB / 57.2MB 13
a3a1cd1d9402 cli.tmg.stvgd.com 0.00% 564KiB / 7.638GiB 0.01% 109kB / 3.61MB 0B / 0B 1
7b3472db2878 couchdb.peer0.asampaio.stvgd.com 0.57% 48.36MiB / 7.638GiB 0.62% 12.2MB / 18.6MB 0B / 3.65MB 38
e7f8a6f4d2af orderer0.ordering-service.inovafil.stvgd.com 0.47% 29.43MiB / 7.638GiB 0.38% 855MB / 858MB 0B / 1.27MB 9
763d30ad33c7 orderer0.ordering-service.tintex.stvgd.com 0.57% 25.91MiB / 7.638GiB 0.33% 860MB / 862MB 0B / 1.24MB 9
4093d41cfe1b ca.tintex.stvgd.com 0.00% 5.867MiB / 7.638GiB 0.08% 13.4kB / 142B 0B / 311kB 6
0d8c0f922391 orderer0.ordering-service.tmg.stvgd.com 0.28% 23.19MiB / 7.638GiB 0.30% 356MB / 354MB 0B / 1.28MB 8
0e4d455e5aed couchdb.peer0.inovafil.stvgd.com 2.17% 48.7MiB / 7.638GiB 0.62% 11.8MB / 18.6MB 3.89MB / 3.58MB 38
aff00e35cae9 cli.tintex.stvgd.com 0.00% 564KiB / 7.638GiB 0.01% 106kB / 3.61MB 0B / 0B 1
48f0293fdf2b orderer0.ordering-service.asampaio.stvgd.com 0.32% 23.92MiB / 7.638GiB 0.31% 358MB / 356MB 0B / 1.3MB 9
a302eb605979 couchdb.peer0.domingosdesousa.stvgd.com 0.55% 49.39MiB / 7.638GiB 0.63% 11.9MB / 18.6MB 0B / 3.24MB 38
705cdfbf643b couchdb.peer0.tintex.stvgd.com 0.77% 47.34MiB / 7.638GiB 0.61% 11.7MB / 18.6MB 0B / 3.28MB 38
c12860ea623c ca.inovafil.stvgd.com 0.00% 5.922MiB / 7.638GiB 0.08% 22.8kB / 27kB 0B / 569kB 6
5f8f1e69c5bd ca.tmg.stvgd.com 0.00% 5.93MiB / 7.638GiB 0.08% 38.9kB / 34.2kB 0B / 307kB 6
1d3761224ee1 fablo-rest.inovafil.stvgd.com 0.00% 41.61MiB / 7.638GiB 0.53% 7.92MB / 8.09MB 0B / 0B 11
b85a80ae8595 cli.asampaio.stvgd.com 0.00% 556KiB / 7.638GiB 0.01% 104kB / 3.61MB 0B / 0B 1
edb3f6c5fe73 peer0.inovafil.stvgd.com 4.57% 78.73MiB / 7.638GiB 1.01% 625MB / 1.2GB 0B / 1.58MB 9
6404e30c1ac3 peer0.asampaio.stvgd.com 1.89% 81.43MiB / 7.638GiB 1.04% 625MB / 1.2GB 0B / 1.72MB 9
0b7dfa9f11fe fablo-rest.tintex.stvgd.com 0.00% 33.52MiB / 7.638GiB 0.43% 12.7kB / 142B 0B / 0B 7
344a4390f1a4 peer0.tmg.stvgd.com 2.20% 89.45MiB / 7.638GiB 1.14% 624MB / 1.2GB 0B / 1.62MB 9
863d54334d0c couchdb.peer0.tmg.stvgd.com 0.61% 48.23MiB / 7.638GiB 0.62% 11.8MB / 18.6MB 0B / 3.08MB 38
5f992dffc3fe fablo-rest.asampaio.stvgd.com 0.00% 41.9MiB / 7.638GiB 0.54% 8.1MB / 8.16MB 0B / 0B 11
f81d9913d221 orderer0.ordering-service.domingosdesousa.stvgd.com 0.32% 20.86MiB / 7.638GiB 0.27% 360MB / 358MB 0B / 1.26MB 9
a8cccdcd3da7 fablo-rest.tmg.stvgd.com 0.00% 34.09MiB / 7.638GiB 0.44% 11.5kB / 142B 0B / 0B 7
0ef9116e9a29 ca.domingosdesousa.stvgd.com 0.00% 5.922MiB / 7.638GiB 0.08% 14.7kB / 1.17kB 0B / 324kB 6
2976324ffc3e cli.domingosdesousa.stvgd.com 0.00% 556KiB / 7.638GiB 0.01% 111kB / 3.61MB 0B / 0B 1
8ac7f9881666 ca.asampaio.stvgd.com 0.00% 5.918MiB / 7.638GiB 0.08% 31.2kB / 53kB 0B / 1.2MB 6
fc0f239cfe44 peer0.domingosdesousa.stvgd.com 1.82% 89.14MiB / 7.638GiB 1.14% 628MB / 1.21GB 0B / 1.86MB 8
2dbe3b898482 fablo-rest.domingosdesousa.stvgd.com 0.00% 34.12MiB / 7.638GiB 0.44% 11.3kB / 0B 0B / 0B 7
62a62e6baf37 peer0.tintex.stvgd.com 1.92% 88.8MiB / 7.638GiB 1.14% 625MB / 1.2GB 0B / 1.76MB 8
48ad2c8e3b46 cli.inovafil.stvgd.com 0.00% 37.04MiB / 7.638GiB 0.47% 170kB / 3.67MB 72.7MB / 0B 1