aries-cloudagent-python feat: Integrate AnonCreds with W3C VCDI Format Support in ACA-Py

Summary

Pull request introduces comprehensive support for AnonCreds within the W3C VCDI formats. The changes include the addition of new models, methods, and protocols that enable the Aries Cloud Agent Python to issue, hold, and verify credentials in alignment with W3C's VCDI standards.

Changes Overview

AnonCreds Integration: Updated the anoncreds module to handle VCDI-compliant credential offers, requests, and credentials.
Indy Models: Adapted indy/models to map onto VCDI credential.
Issue Credential Protocol v2.0: Incorporated VCDI support into the existing issue_credential protocol
VCDI Credential Format Handling: Introduced a new vc_di module within formats to handle the specificities of the VCDI credential format.
Updated alice/faber demo: Introduced handling of multiple cred demo with cred-type argument

Detailed Changes

Added handler.py in protocols/issue_credential/v2_0/formats/vc_di to manage VCDI credential operations.
Created test_handler.py in the same directory to ensure robustness and reliability through comprehensive testing.
Introduced vc_di.py within models/detail for detailed VCDI model definitions.
Updated test_routes.py to test the integration of VCDI routes within the agent's service layer.
Revised routes.py to handle API endpoints related to VCDI credential operations.
Updated faber.py
Update alice.py
Updated agent_container.py
Updated performance.py

Additional Notes

Link to the detailed documentation and specifications: W3C VCDI Integration

Mar 31 '24 14:03 sarthakvijayvergiya

Does this replace the other PR? (In which case we can close it)

Also please make sure to include in the description of this PR a ... description of what the PR includes (and if there is any work still in progress) to help us to know what to review.

Thanks

Mar 31 '24 15:03 ianco

Does this replace the other PR? (In which case we can close it)

Also please make sure to include in the description of this PR a ... description of what the PR includes (and if there is any work still in progress) to help us to know what to review.

Thanks

Thank you for your comments. Indeed, this replaces the previous PR and will close that one. I will update the detaild description as soon as I complete the modification and testing.

Mar 31 '24 22:03 sarthakvijayvergiya

Looks good @sarthakvijayvergiya make sure as you wrap up to check all the comments in https://github.com/hyperledger/aries-cloudagent-python/pull/2830 as well

Apr 01 '24 13:04 gvelez17

Does this replace the other PR? (In which case we can close it)

Also please make sure to include in the description of this PR a ... description of what the PR includes (and if there is any work still in progress) to help us to know what to review.

Thanks

yes, closed the other. this one is not fully ready for review yet, @sarthakvijayvergiya however can you keep the description up to date on what parts are complete?

Apr 01 '24 13:04 gvelez17

Also ... I think for the demo we should add an option where the user can change Faber's credential_type "on the fly" (i.e. specify on startup but then change while faber is running). Eventually we need to test issuing a credential using "vc_di" and then testing a "legacy" proof request, or issuing a "legacy" credential and then requesting a "vc_di" proof.

Again not necessary for this PR ...

Apr 02 '24 17:04 ianco

I'm still doing some testing, but overall looks good! Just a few minor comments ...

A couple of things we need:

updates to documentation - a reference in the demo docs that --cred-type vc_di is now an option
maybe in docs/features ... reference to vc_di as a supported format for issuing credentials (@swcurran where should this be documented?)
at least one integration test

For the last one, it should be a simple matter of adding a new scenario to 0453-issue-credential.feature, like:

    @GHA @WalletType_Askar_AnonCreds @BasicTest
    Examples:
       | Acme_capabilities                          | Bob_capabilities              | Schema_name    | Credential_data          | Acme_extra | Bob_extra |
       | --public-did --wallet-type askar-anoncreds | --wallet-type askar-anoncreds | driverslicense | Data_DL_NormalizedValues |            |           |
       | --public-did --wallet-type askar-anoncreds --cred-type vc_di | --wallet-type askar-anoncreds | driverslicense | Data_DL_NormalizedValues |            |           |

Apr 04 '24 15:04 ianco

I would really like to see at least an interactive run of using Credo-TS as a holder, and using ACA-Py as an issuer and verifier. Crucial for the CWU is to see the interop. Thoughts on that? We can engage the devs from Animo in helping with that.

Apr 04 '24 16:04 swcurran

I would really like to see at least an interactive run of using Credo-TS as a holder, and using ACA-Py as an issuer and verifier. Crucial for the CWU is to see the interop. Thoughts on that? We can engage the devs from Animo in helping with that.

Could we do this using AATH? (Or AMTH?)

Apr 04 '24 16:04 ianco

AATH would be awesome. I don’t know the effort, but we would need someone that knows AATH to help out. The scope is too much for this work, unfortunately. My big concern is the current state of the Credo Backchannel. It needs work.

Apr 04 '24 16:04 swcurran

I ran a quick test of the alice/faber demo and it failed. I did the ./run_demo build ... on each agent and then:

Faber: $ AGENT_PORT_OVERRIDE=8010 ./run_demo run faber --wallet-type askar-anoncreds --cred-type vc_di

Alice: $ ./run_demo run alice --wallet-type askar-anoncreds

Then connected and tried to issue a credential:

[1/2/3/4/T/X] 1                                                                                                                                                                                                    

#13 Issue credential offer to X
Faber      | Credential: state = offer-sent, cred_ex_id = d5d5afa6-bb30-4b35-a085-96b935cbb9cc
Faber      | 2024-04-29 16:19:42,627 aries_cloudagent.messaging.models.base ERROR V20CredRequest message validation error:
Faber      | Traceback (most recent call last):
Faber      |   File "/home/aries/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
Faber      |     schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
Faber      |     return self._do_load(
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
Faber      |     raise exc
Faber      | marshmallow.exceptions.ValidationError: {'binding_proof': {'anoncreds_link_secret': {'entropy': ['Value did:peer:4zQmetubuCeDizQRVaJWscDcGxrz65Cyv7RyFMAmCCFpQk22 is not an indy decentralized identifier (DID)']}}}
Faber      | 2024-04-29 16:19:42,630 aries_cloudagent.core.dispatcher ERROR Message parsing failed: Error deserializing message: V20CredRequest schema validation failed, sending problem report

Tagging @swcurran and @dbluhm because it looks like we're using a did:peer:4 here for some reason and the marshmallow schema isn't accepting it ...

Apr 29 '24 16:04 ianco

... also when I run the integration tests they all report as "pass" but when I look through the logs there are errors:

$ AGENT_PORT_OVERRIDE=8030 ./run_bdd -t @T003-RFC0453
...
    When "Bob" requests a credential with data Data_DL_NormalizedValues from "Acme" it fails # features/steps/0453-issue-credential.py:127
Bob.agent  | 2024-04-29 16:41:48,606 aries_cloudagent.admin.server ERROR Handler error with exception: Unprocessable Entity
Bob.agent  | Traceback (most recent call last):
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/asyncparser.py", line 90, in parse
Bob.agent  |     result = schema.load(parsed)
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
>>> after_scenario activated
>>> shutting down active agents ...
    shutting down: Acme
Shutting down agent ...
Bob.agent  |     return self._do_load(
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
Bob.agent  |     raise exc
Bob.agent  | marshmallow.exceptions.ValidationError: {'filter': {'ld_proof': ['Missing data for required field.']}}
Bob.agent  | 
Bob.agent  | During handling of the above exception, another exception occurred:
Bob.agent  | 
Bob.agent  | Traceback (most recent call last):
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 181, in ready_middleware
Bob.agent  |     return await handler(request)
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 218, in debug_middleware
Bob.agent  |     return await handler(request)
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 386, in check_multitenant_authorization
Bob.agent  |     return await handler(request)
Bob.agent  |   File "/home/aries/aries_cloudagent/admin/server.py", line 451, in setup_context
Bob.agent  |     return await task
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/futures.py", line 284, in __await__
Bob.agent  |     yield self  # This tells Task to wait for completion.
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 328, in __wakeup
Bob.agent  |     future.result()
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/futures.py", line 201, in result
Bob.agent  |     raise self._exception
Bob.agent  |   File "/usr/local/lib/python3.9/asyncio/tasks.py", line 256, in __step
Bob.agent  |     result = coro.send(None)
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/aiohttp_apispec/middlewares.py", line 33, in validation_middleware
Bob.agent  |     data = await request.app["_apispec_parser"].parse(
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/asyncparser.py", line 94, in parse
Bob.agent  |     await self._on_validation_error(
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/asyncparser.py", line 108, in _on_validation_error
Bob.agent  |     await error_handler(error, req, schema, error_status_code, error_headers)
Bob.agent  |   File "/home/aries/.local/lib/python3.9/site-packages/webargs/aiohttpparser.py", line 163, in handle_error
Bob.agent  |     raise error_class(
Bob.agent  | aiohttp.web_exceptions.HTTPUnprocessableEntity: Unprocessable Entity

Apr 29 '24 16:04 ianco

Running the alice/faber demo with the --events flag I can see that alice receives the credential offer and sends the credential request, and faber gets the error receiving the request. Here is the cred request, I'm not sure how the did:peer:4 gets in there or why faber cares what is the format of the "entropy":

        "cred_request": {
            "vc_di": {
                "data_model_version": "2.0",
                "binding_proof": {
                    "anoncreds_link_secret": {
                        "entropy": "did:peer:4zQmSz9RPHAXYyKe8RNEaH3DWz6jAzYVADZWt82h88URnYTg",
                        "cred_def_id": "7JcY3HVEaUwXdoXeSeFSSi:3:CL:176:faber.agent.degree_schema",
                        "blinded_ms": {
                            "u": "98157803620660872170402354543798974200462544452245591862217725500504386281160961655563679393987978770632568196480064042932031478725476063915670091399243123044122923246815807721802098890962420986125428117490105853961326051702005632977344987905700321953571139659140865989527824169498631840709995380507817419826662772277722439467289687048146883920558627162230071386049697776277292986334401664583542979134445423113364037510563284891531520997746722867719148651795256773203744802404879619935812531852155599459431918862100233592485693666170250989538356501445054894228211868579282658727052977550795345340387507015924540012548",
                            "ur": null,
                            "hidden_attributes": [
                                "master_secret"
                            ],
                            "committed_attributes": {}
                        },
                        "blinded_ms_correctness_proof": {
                            "c": "59433802809596338271128470279603183486702814236484143165828684066161975073261",
                            "v_dash_cap": "2291688135408765057454102999713290079468470806026404345010878504526654291637245987562467966244405580205292137949428093919085198310059447159324212870553877066735257526420053820932460859921179097316828804442130780534924326501078304785117073706502754035170388518062719574873475329608590008271662112168037204411383802356689765012068442072241086717499759364113588149218843455030220001906292019130107467050776769753277284854618996023764841503608868017026452558261583504715139095668493430196979432551095789861656950035970995288498080778896408155269698695357277285620211683342360165406497987029591209902977936516027482604268823342975636845320416001178746420147803174461964012518294044516186535247995738909218895238190583877815",
                            "m_caps": {
                                "master_secret": "22460859598887235898916958953680270931050220445132084890048716866094989991233598532889415785637083539883931553700153352400740539022798084603405103364810530827905738895125862866024"
                            },
                            "r_caps": {}
                        },
                        "nonce": "595869006883776147466000"
                    },
                    "didcomm_signed_attachment": {
                        "attachment_id": "test"
                    }
                }
            }
        }

Apr 29 '24 17:04 ianco

I ran a quick test of the alice/faber demo and it failed. I did the ./run_demo build ... on each agent and then:

Faber: $ AGENT_PORT_OVERRIDE=8010 ./run_demo run faber --wallet-type askar-anoncreds --cred-type vc_di

Alice: $ ./run_demo run alice --wallet-type askar-anoncreds

Then connected and tried to issue a credential:

[1/2/3/4/T/X] 1                                                                                                                                                                                                    

#13 Issue credential offer to X
Faber      | Credential: state = offer-sent, cred_ex_id = d5d5afa6-bb30-4b35-a085-96b935cbb9cc
Faber      | 2024-04-29 16:19:42,627 aries_cloudagent.messaging.models.base ERROR V20CredRequest message validation error:
Faber      | Traceback (most recent call last):
Faber      |   File "/home/aries/aries_cloudagent/messaging/models/base.py", line 196, in deserialize
Faber      |     schema.loads(obj) if isinstance(obj, str) else schema.load(obj),
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 723, in load
Faber      |     return self._do_load(
Faber      |   File "/home/aries/.local/lib/python3.9/site-packages/marshmallow/schema.py", line 910, in _do_load
Faber      |     raise exc
Faber      | marshmallow.exceptions.ValidationError: {'binding_proof': {'anoncreds_link_secret': {'entropy': ['Value did:peer:4zQmetubuCeDizQRVaJWscDcGxrz65Cyv7RyFMAmCCFpQk22 is not an indy decentralized identifier (DID)']}}}
Faber      | 2024-04-29 16:19:42,630 aries_cloudagent.core.dispatcher ERROR Message parsing failed: Error deserializing message: V20CredRequest schema validation failed, sending problem report

Tagging @swcurran and @dbluhm because it looks like we're using a did:peer:4 here for some reason and the marshmallow schema isn't accepting it ...

I created issue #2923 for this — I think it is an urgent one for 0.12.1. An qualified DID is being used by Alice, being sent in the request message, and is incorrectly being verified and required to be an unqualified (or did:sov) peer DID. The verification needs to be removed from wherever it is happening.

Apr 29 '24 17:04 swcurran

I created issue #2923 for this — I think it is an urgent one for 0.12.1. An qualified DID is being used by Alice, being sent in the request message, and is incorrectly being verified and required to be an unqualified (or did:sov) peer DID. The verification needs to be removed from wherever it is happening.

I suspect this is just an issue with the new vc_di but I will do a bit of digging

Apr 29 '24 17:04 ianco

@ianco when you take a look at this, my thought is if the code is correct and passing the current integration tests, we should merge this PR before it gets more outdated, and start a new one for any additional integration tests and also the one for the DIF format (see questions there)

Let me know if you think differently.

May 06 '24 16:05 gvelez17

One question - if in the DIF pr they are calling it di_vc should we change all references in here to di_vc ? @swcurran was there a decision made about what the name is, of Veriable Credentials with Data Integrity, 'vc_di' or 'di_vc' ?

May 06 '24 16:05 gvelez17

@ianco when you take a look at this, my thought is if the code is correct and passing the current integration tests, we should merge this PR before it gets more outdated, and start a new one for any additional integration tests and also the one for the DIF format (see questions there)

Let me know if you think differently.

Yes once this is working we should merge, and then we can deal with outstanding functionality in separate PR's.

May 06 '24 17:05 ianco