Discovery service Peer TLS CA Cert problem

[scmanjarrez]

Hi, When I use fabric client discovery service, a gRPC channel is created for every peer in network using wrong TLS CA Cert because it is base64 encoded.

If I query Peer information, I get the following output:

[Peer:_name=peer0.org2.example.com,_lock=<unlocked _thread.RLock object owner=0 count=0 at 0x7fbf370bc780>,_channels=[],_endpoint=peer0.org2.example.com:9051,_grpc_options={'grpc.ssl_target_name_override': 'peer0.org2.example.com'},_ssl_target_name=peer0.org2.example.com,_tls_ca_certs_path=b'LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSUNWekNDQWYyZ0F3SUJBZ0lRY0RHeWRDVnVrSUpqekNsWUlmcHltekFLQmdncWhrak9QUVFEQWpCMk1Rc3cKQ1FZRFZRUUdFd0pWVXpFVE1CRUdBMVVFQ0JNS1EyRnNhV1p2Y201cFlURVdNQlFHQTFVRUJ4TU5VMkZ1SUVaeQpZVzVqYVhOamJ6RVpNQmNHQTFVRUNoTVFiM0puTWk1bGVHRnRjR3hsTG1OdmJURWZNQjBHQTFVRUF4TVdkR3h6ClkyRXViM0puTWk1bGVHRnRjR3hsTG1OdmJUQWVGdzB5TURBMk1Ua3hNelU1TURCYUZ3MHpNREEyTVRjeE16VTUKTURCYU1IWXhDekFKQmdOVkJBWVRBbFZUTVJNd0VRWURWUVFJRXdwRFlXeHBabTl5Ym1saE1SWXdGQVlEVlFRSApFdzFUWVc0Z1JuSmhibU5wYzJOdk1Sa3dGd1lEVlFRS0V4QnZjbWN5TG1WNFlXMXdiR1V1WTI5dE1SOHdIUVlEClZRUURFeFowYkhOallTNXZjbWN5TG1WNFlXMXdiR1V1WTI5dE1Ga3dFd1lIS29aSXpqMENBUVlJS29aSXpqMEQKQVFjRFFnQUVrRGZXYmJBV0FadURvSWRGSUZhZFdmVWEyTldvVWhhWVNlTjJRZDBWUjRQMVhqRDJKWXEvR0QxZQpqb3pWajBNSzZNaVZraGI5aUNJdmw5Ynp5L2FvSmFOdE1Hc3dEZ1lEVlIwUEFRSC9CQVFEQWdHbU1CMEdBMVVkCkpRUVdNQlFHQ0NzR0FRVUZCd01DQmdnckJnRUZCUWNEQVRBUEJnTlZIUk1CQWY4RUJUQURBUUgvTUNrR0ExVWQKRGdRaUJDQkNhYjBWQXR6VjJVd2pXYXRyK0F0VkZCeXgwTXV5Y29Oem5mYmNMNDErOFRBS0JnZ3Foa2pPUFFRRApBZ05JQURCRkFpQUNCVlMyTWlvWlE5RzdTODRLZWlJSGh2cVM1N3Y3akxtQmVwYlVKeTNBMEFJaEFQakwrbUR4CmZNc3NiRk0zWk5VMThhZlkxTWdObkFEeGRkZ20vcTFoZnZwbAotLS0tLUVORCBDRVJUSUZJQ0FURS0tLS0tCg==',_client_key_path=None,_client_cert_path=None,_channel=<aiogrpc.channel.Channel object at 0x7fbf37077828>,_endorser_client=<hfc.protos.peer.peer_pb2_grpc.EndorserStub object at 0x7fbf37077208>,_discovery_client=<hfc.protos.discovery.protocol_pb2_grpc.DiscoveryStub object at 0x7fbf370774a8>,_event_client=<hfc.protos.peer.events_pb2_grpc.DeliverStub object at 0x7fbf37077550>]

The important information is in _tls_ca_certs_path, where is the TLS CA Cert in PEM format but encoded, so when util/channel:create_grpc_channel is called, it fails generating the secure channel.

    if cert_file:
        if isinstance(cert_file, bytes):
            root_cert = cert_file
        ...

E0619 17:19:57.225199736    5156 ssl_transport_security.cc:738] Could not load any root certificate.
E0619 17:19:57.225230344    5156 ssl_transport_security.cc:1837] Cannot load server root certificates.
E0619 17:19:57.225251080    5156 ssl_security_connector.cc:113] Handshaker factory creation failed with TSI_INVALID_ARGUMENT.
E0619 17:19:57.225260517    5156 secure_channel_create.cc:132] Failed to create secure subchannel for secure name 'peer0.org2.example.com:9051'
E0619 17:19:57.225270501    5156 secure_channel_create.cc:50] Failed to create channel args during subchannel creation.

Decoding the bytes in util/channel.py:create_grpc_channel fixed the problem. However, I'm not sure what are the consequences in other functions.

    if cert_file:
        if isinstance(cert_file, bytes):
            root_cert = base64.b64decode(cert_file)
        ...

Edit: I have checked and it also happens when orderer TLS CA Cert is loaded with discovery.

[yeasy]

I guess the question is because of using base64 encoded cert file. Certainly you can try to decode the cert file if the direct reading fails.

[scmanjarrez]

I can't decode it myself because client.py:init_with_discovery create the orderer/peer gRPC channel in utils/channel.py:create_grpc_channel with the discovered data.

client.py and orderer.py

orderer = Orderer(name=orderer_info['host'],
                                  endpoint=orderer_endpoint)
                orderer.init_with_bundle(info)

 self._tls_ca_certs_path = info['tlsCACerts']['path']
...
self._channel = create_grpc_channel(
                self._endpoint,
                self._tls_ca_certs_path,
                self._client_key_path,
                self._client_cert_path,
                opts=[(opt, value) for opt, value in
                      self._grpc_options.items()])
self._orderer_client = ab_pb2_grpc.AtomicBroadcastStub(
                self._channel)

client.py and peer.py

peer = Peer(name=target_name)
                peer.init_with_bundle(info)

 self._tls_ca_certs_path = info['tlsCACerts']['path']
...
self._channel = create_grpc_channel(
                self._endpoint,
                self._tls_ca_certs_path,
                self._client_key_path,
                self._client_cert_path,
                opts=[(opt, value) for opt, value in
                      self._grpc_options.items()])
self._endorser_client = peer_pb2_grpc.EndorserStub(self._channel)

I can't decode the certs manually and pass to the client/peer/orderder without modifying all the code in client.py, or peer.py and orderer.py.

[scmanjarrez]

Any update with this issue? I can send a PR with the changes needed, but I don't know which parts of the code use encoded certs and can break.