hyperledger
Broken Pipe Error Between Peer and external Chaincode.
Description
Hello,
When i try to invoke chaincode with below command. I am getting Broken Pipe error in Peer logs.
When the Peer try to connect with Chaincode, TLS handshake happens and then I get broken pipe error.
The Setup is deployed on OCP cluster (Kuberenetes v1.27). There is no connectivity issues as they are in same network.
I have also created Headless service for chaincode deployment.
ERROR: 2024-07-15 09:04:35.602 UTC [grpc] WarningDepth -> DEBU 38e [core] [Channel #134 SubChannel #135] grpc: addrConn.createTransport failed to connect to { "Addr": "chaincode1-service:9000", "ServerName": "chaincode1-service:9000", "Attributes": null, "BalancerAttributes": null, "Type": 0, "Metadata": null }. Err: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 38f [core] [Channel #134 SubChannel #135] Subchannel Connectivity change to TRANSIENT_FAILURE 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 390 [core] pickfirstBalancer: UpdateSubConnState: 0xc00227c140, {TRANSIENT_FAILURE write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe} 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 391 [core] [Channel #134] Channel Connectivity change to TRANSIENT_FAILURE 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 392 [core] [Channel #134] Channel Connectivity change to SHUTDOWN 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 393 [core] [Channel #134 SubChannel #135] Subchannel Connectivity change to SHUTDOWN 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 394 [core] [Channel #134 SubChannel #135] Subchannel deleted 2024-07-15 09:04:35.602 UTC [grpc] InfoDepth -> DEBU 395 [core] [Channel #134] Channel deleted 2024-07-15 09:04:35.602 UTC 38e6 DEBU [chaincode] Launch -> stopping due to error while launching: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe failed to create new connection github.com/hyperledger/fabric/internal/pkg/comm.ClientConfig.Dial /internal/pkg/comm/config.go:151 github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).createConnection /core/chaincode/extcc/extcc_handler.go:33 github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).Stream /core/chaincode/extcc/extcc_handler.go:45 github.com/hyperledger/fabric/core/chaincode.(*RuntimeLauncher).Launch.func1 /core/chaincode/runtime_launcher.go:90 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 error creating grpc connection to chaincode1-service:9000 error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed 2024-07-15 09:04:35.602 UTC 38e7 DEBU [chaincode] Launch -> launch complete 2024-07-15 09:04:35.602 UTC 38e8 DEBU [chaincode] Deregister -> deregister handler: asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 2024-07-15 09:04:35.602 UTC 38e9 INFO [endorser] callChaincode -> finished chaincode: asset-transfer-basic_9 duration: 5ms channel=mychannel1 txID=ffb61445 2024-07-15 09:04:35.602 UTC 38ea ERRO [endorser] simulateProposal -> failed to invoke chaincode asset-transfer-basic_9, error: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe failed to create new connection github.com/hyperledger/fabric/internal/pkg/comm.ClientConfig.Dial /internal/pkg/comm/config.go:151 github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).createConnection /core/chaincode/extcc/extcc_handler.go:33 github.com/hyperledger/fabric/core/chaincode/extcc.(*ExternalChaincodeRuntime).Stream /core/chaincode/extcc/extcc_handler.go:45 github.com/hyperledger/fabric/core/chaincode.(*RuntimeLauncher).Launch.func1 /core/chaincode/runtime_launcher.go:90 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 error creating grpc connection to chaincode1-service:9000 error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed could not launch chaincode asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Launch /core/chaincode/chaincode_support.go:87 github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Invoke /core/chaincode/chaincode_support.go:203 github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute /core/chaincode/chaincode_support.go:161 github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute /core/endorser/support.go:126 github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode /core/endorser/endorser.go:120 github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal /core/endorser/endorser.go:187 github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError /core/endorser/endorser.go:409 github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal /core/endorser/endorser.go:350 github.com/hyperledger/fabric/internal/pkg/gateway.(*EndorserServerAdapter).ProcessProposal /internal/pkg/gateway/gateway.go:41 github.com/hyperledger/fabric/internal/pkg/gateway.(*Server).Evaluate.func1 /internal/pkg/gateway/evaluate.go:64 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 failed to execute transaction ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4 github.com/hyperledger/fabric/core/chaincode.processChaincodeExecutionResult /core/chaincode/chaincode_support.go:167 github.com/hyperledger/fabric/core/chaincode.(*ChaincodeSupport).Execute /core/chaincode/chaincode_support.go:162 github.com/hyperledger/fabric/core/endorser.(*SupportImpl).Execute /core/endorser/support.go:126 github.com/hyperledger/fabric/core/endorser.(*Endorser).callChaincode /core/endorser/endorser.go:120 github.com/hyperledger/fabric/core/endorser.(*Endorser).simulateProposal /core/endorser/endorser.go:187 github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposalSuccessfullyOrError /core/endorser/endorser.go:409 github.com/hyperledger/fabric/core/endorser.(*Endorser).ProcessProposal /core/endorser/endorser.go:350 github.com/hyperledger/fabric/internal/pkg/gateway.(*EndorserServerAdapter).ProcessProposal /internal/pkg/gateway/gateway.go:41 github.com/hyperledger/fabric/internal/pkg/gateway.(*Server).Evaluate.func1 /internal/pkg/gateway/evaluate.go:64 runtime.goexit /usr/local/go/src/runtime/asm_amd64.s:1598 channel=mychannel1 txID=ffb61445 2024-07-15 09:04:35.602 UTC 38eb DEBU [lockbasedtxmgr] Done -> Done with transaction simulation / query execution [ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4] 2024-07-15 09:04:35.602 UTC 38ec WARN [endorser] ProcessProposal -> Failed to invoke chaincode channel=mychannel1 chaincode=asset-transfer-basic_9 error="error in simulation: failed to execute transaction ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4: could not launch chaincode asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed: error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: error creating grpc connection to chaincode1-service:9000: failed to create new connection: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe" 2024-07-15 09:04:35.602 UTC 38ed DEBU [gateway] func1 -> Evaluate call to endorser failed chaincode=asset-transfer-basic_9 channel=mychannel1 txID=ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4 endorserAddress=fabric-peer1-service:7061 endorserMspid=Org6MSP error="error in simulation: failed to execute transaction ffb614459129997724ec2e805100d8a063ed8d7f464a981c63fb3f7dc57cadf4: could not launch chaincode asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: connection to asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5 failed: error cannot create connection for asset-transfer-basic_9:8fda7b2f506305775ba682399d4a421f7d598ce98ce198734b86a2a72c8d6fb5: error creating grpc connection to chaincode1-service:9000: failed to create new connection: write tcp 10.244.6.106:56446->10.244.7.203:9000: write: broken pipe" 2024-07-15 09:04:35.602 UTC 38ee INFO [comm.grpc.server] 1 -> unary call completed grpc.service=gateway.Gateway grpc.method=Evaluate grpc.request_deadline=2024-07-15T09:06:35.573Z grpc.peer_address=10.244.6.110:45798 error="rpc error: code = Aborted desc = failed to evaluate transaction, see attached details for more info" grpc.code=Aborted grpc.call_duration=7.12576ms 2024-07-15 09:04:35.613 UTC [grpc] InfoDepth -> DEBU 396 [transport] transport: loopyWriter.run returning. connection error: desc = "transport is closing
Steps to reproduce
No response
the resason of " write: broken pipe“ is alawys two points: 1.The server's host has exceeded the maximum number of connections. 2.The client disconnects before receiving a response from the server.
For the "Broken Pipe" error, you can take the following solutions:
Gracefully Handle Connection Closures: When writing network applications, ensure that you can gracefully handle connection closures. For example, catch "Broken Pipe" exceptions when attempting to write data and appropriately clean up resources (such as closing sockets). Add Retry Logic: For "Broken Pipe" errors that may be caused by network issues, add retry logic to your application. This can help recover communication after brief network interruptions. Adjust Timeout and Idle Connection Settings: If the issue is due to timeouts or idle connection closures, try adjusting relevant system or application settings. For example, increase connection timeout times or disable idle connection closure features. Check System Resources: Ensure that system resources (such as the number of file descriptors) are sufficient and have not reached their limits. If resources are insufficient, you may need to increase resource limits or optimize the application's resource usage. Log and Analyze Logs: By logging and analyzing the logs of network applications, you can better understand the circumstances and causes of "Broken Pipe" errors. This helps identify and resolve underlying issues. Use a More Stable Network: If the issue is due to an unstable network, consider using a more stable network connection or adding network redundancy to improve the reliability of communications.