[Confidential Ledger] Add the Confidential Ledger Receipt Verification Client Library

[andpiccione]

Description

Related issue: https://github.com/Azure/azure-sdk/issues/5109

This PR adds a new client library for Azure Confidential Ledger inside the existing Data Plane SDK that can be used by customers to verify write transaction receipts. For context, this library is supposed to be used to verify receipts issued by a Confidential Ledger instance for committed transactions without connecting to the ledger or to any other service: this would provide Confidential Ledgers users with a simple tool to verify receipts offline, without the need to trust code that doesn't run in their client machines.

This client library is not based on REST API / swagger specs and it is supposed to be an ad-hoc, specific offline library for Azure Confidential Ledger receipts verification. We discussed with @johanste a few months ago regarding the best approach to deliver such specific tool for our customers and we concluded that including it as part of the SDKs would be the best approach given that a developer use-case for receipt verification usage is in scope. Given the unicity of such a library, we are open to any suggestions for better modelling and delivering this tool to our customers.

In this first version, the library includes:

• Code for the receipt verification algorithm, along with models, and internal utilities for serializing/deserializing models (this one was copy-pasted from an auto-generated REST API library)
• Tests for the main models used and the receipt verification algorithm in different test cases
• Sample script that can be used by customers to write a transaction to a running Confidential Ledger, fetch a write receipt for the same transaction and verify its contents using the library.
• Update to other files inside the SDK (README, CHANGELOG, etc.)

All SDK Contribution checklist:

• [X] The pull request does not introduce [breaking changes]
• [X] CHANGELOG is updated for new features, bug fixes or other significant changes.
• [X] I have read the contribution guidelines.

General Guidelines and Best Practices

• [X] Title of the pull request is clear and informative.
• [X] There are a small number of commits, each of which have an informative message. This means that previously merged commits do not appear in the history of the PR. For more information on cleaning up the commits in your PR, see this page.

Testing Guidelines

• [X] Pull request includes test coverage for the included changes.

[andpiccione]

@johanste I am tagging you as reviewer since we discussed about the addition of this library a few months back in an email thread. Please let me know if you have any questions or if anything needs further clarification.

[azure-sdk]

API change check

APIView has identified API level changes in this PR and created following API reviews.

azure-confidentialledgertools-receiptverification azure-confidentialledger

[andpiccione]

FYI, the CI tests are consistently failing because of another test (TestConfidentialLedgerClient.test_tls_cert_convenience_aad_user) in the separate azure-confidentialledger library (which does not depend on or is not a dependent of the new added azure-confidentialledgertools-receiptverification libraries): see test failure.

I didn't modify the azure-confidentialledger library and I'm not exactly sure why it suddenly started failing. I will investigate this more at a later point, but I just wanted to point out that the tests for the new library are passing and work as expected.

[lynshi]

FYI, the CI tests are consistently failing because of another test (TestConfidentialLedgerClient.test_tls_cert_convenience_aad_user) in the separate azure-confidentialledger library (which does not depend on or is not a dependent of the new added azure-confidentialledgertools-receiptverification libraries): see test failure.

I didn't modify the azure-confidentialledger library and I'm not exactly sure why it suddenly started failing. I will investigate this more at a later point, but I just wanted to point out that the tests for the new library are passing and work as expected.

Example error message

self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x0000000003a7ad08>

    def get_event_loop(self):
        """Get the event loop for the current context.
    
        Returns an instance of EventLoop or raises an exception.
        """
        if (self._local._loop is None and
                not self._local._set_called and
                isinstance(threading.current_thread(), threading._MainThread)):
            self.set_event_loop(self.new_event_loop())
    
        if self._local._loop is None:
            raise RuntimeError('There is no current event loop in thread %r.'
>                              % threading.current_thread().name)
E           RuntimeError: There is no current event loop in thread 'MainThread'.

/opt/hostedtoolcache/PyPy/3.7.13/x64/lib-python/3/asyncio/events.py:644: RuntimeError

ERROR: InvocationError for command /mnt/vss/_work/1/s/sdk/confidentialledger/azure-confidentialledger/.tox/whl/bin/pytest --junitxml=/mnt/vss/_work/1/s/sdk/confidentialledger/azure-confidentialledger/test-junit-whl.xml --verbose --durations=10 --ignore=azure --ignore=.tox --ignore=build --ignore=.eggs --no-cov --ignore azure_confidentialledger.egg-info . (exited with code 1)

My hunch is pipeline flakiness not something wrong with our tests.

[andpiccione]

FYI, the CI tests are consistently failing because of another test (TestConfidentialLedgerClient.test_tls_cert_convenience_aad_user) in the separate azure-confidentialledger library (which does not depend on or is not a dependent of the new added azure-confidentialledgertools-receiptverification libraries): see test failure.

I didn't modify the azure-confidentialledger library and I'm not exactly sure why it suddenly started failing. I will investigate this more at a later point, but I just wanted to point out that the tests for the new library are passing and work as expected.

Example error message

self = <asyncio.unix_events._UnixDefaultEventLoopPolicy object at 0x0000000003a7ad08>



    def get_event_loop(self):

        """Get the event loop for the current context.

    

        Returns an instance of EventLoop or raises an exception.

        """

        if (self._local._loop is None and

                not self._local._set_called and

                isinstance(threading.current_thread(), threading._MainThread)):

            self.set_event_loop(self.new_event_loop())

    

        if self._local._loop is None:

            raise RuntimeError('There is no current event loop in thread %r.'

>                              % threading.current_thread().name)

E           RuntimeError: There is no current event loop in thread 'MainThread'.



/opt/hostedtoolcache/PyPy/3.7.13/x64/lib-python/3/asyncio/events.py:644: RuntimeError

ERROR: InvocationError for command /mnt/vss/_work/1/s/sdk/confidentialledger/azure-confidentialledger/.tox/whl/bin/pytest --junitxml=/mnt/vss/_work/1/s/sdk/confidentialledger/azure-confidentialledger/test-junit-whl.xml --verbose --durations=10 --ignore=azure --ignore=.tox --ignore=build --ignore=.eggs --no-cov --ignore azure_confidentialledger.egg-info . (exited with code 1)

My hunch is pipeline flakiness not something wrong with our tests.

I re-run the pipeline several times in the past days and this test is consistently failing with the same error for a few jobs. I didn't look that carefully, but I believe there must be something wrong with this test because it's failing too many times (assuming this is real flakiness and not something that suddenly changed on our backend).

[johanste]

I am quite concerned about the comments about calling sync code from the __init__ of an async class. In addition to it likely being (at least part of) the problem with the test, it is a real issue when using the client in an async context. Blocking the loop is overall badness.

[lynshi]

I am quite concerned about the comments about calling sync code from the __init__ of an async class. In addition to it likely being (at least part of) the problem with the test, it is a real issue when using the client in an async context. Blocking the loop is overall badness.

I think you are referring to the __init__ of ConfidentialLedgerClient here.

# We'll need to fetch the TLS certificate.
identity_service_client = ConfidentialLedgerCertificateClient(**kwargs)

# Ledger URIs are of the form https://<ledger id>.confidential-ledger.azure.com.
ledger_id = endpoint.replace("https://", "").split(".")[0]
ledger_cert = identity_service_client.get_ledger_identity(ledger_id, **kwargs)

with open(ledger_certificate_path, "w", encoding="utf-8") as outfile:
    outfile.write(ledger_cert["ledgerTlsCertificate"])

We can't have an async constructor in Python, and the network call in the async client __init__ is important for the "auto-magic" setup experience. Some options are a helper method on the object to set up an existing client or a factory pattern. Stack Overflow posts describe other ways as well but there doesn't seem to be one way of doing this. Do you have a recommendation for what would be best?

I think we will go with the factory pattern but customers will have to be a bit more intentional about which initialization flow they want when using the async client.

[johanste]

Either a factory method pattern or to just initiate the request from the __init__ and "join"/retrieve the result in a subsequent call (or delay the call until the value is needed)

[andpiccione]

@annatisch Could you please have another round of review based on my replies to the comments above? Thank you very much in advance!

[andpiccione]

@lmazuel @scbedd @annatisch Could you please merge the PR if everything is ok for you? I'm not authorized to do it.