vc-test-suite
vc-test-suite copied to clipboard
w3c
Evidence extension point (was: Improve tests for Evidence)
See https://www.w3.org/TR/vc-data-model/#evidence
DocumentVerification2018...
this tremendously poorly defined and should be defined for all serializations or removed.
Ideally we would have a test suite that shows how this property supports interoperability, and that suite would cover popular feature such as identity assurance in sufficient detail.
Please do not remove this.
There are some use cases where providing evidence of the pre-issuance DID auth is exceptionally valuable.
This property is meant to be an extension point and we should keep it as such.
@jandrieu can you point to examples of this extension that are being used in the wild today?
I know folks have been considering it, we should gather evidence that it's being used or being considered, and if we can't we should try and remove it to simplify the spec.
I can't, today, speak publicly about either of the two initiatives I know of who are expecting this capability.
What I will say is that without evidence of having performed did-auth embedded in the VC itself, we have no mechanism to verify whether or not the issuer, in fact, proofed control of the DID before issuance. For those subject=holder use cases that rely on the verifier to compare the VP signer to the Subject ID to perform a second step in identity assurance, the evidence property is the only way (within the current spec) that I know of to demonstrate that the first step was completed satisfactorily.
This extension point exists for a reason. Removing it just to "simplify the spec" isn't appropriate here.
I think @selfissued also knows of some folks considering this feature, we should strive to get it covered in the test suite, if it really is being used.
Blocked by an item in the VC extensions directory that defines evidence that we can point to and test against?
It points to the use of the OIDF Identity Assurance draft standard as an example of Evidence
I suggest the following:
- evidence is removed from the core spec and added as an extension to vc-specs-dir.
- evidence is added to the core spec AFTER 2 implementations of the Evidence property have been tested for a given type.
Similar argument would apply to the other properties that are in the core spec, but for which we have no evidence (cough)... of actual implementation or use.
Example from Open Badges
"evidence": [
{
"id": "https://1edtech.edu/credentials/3732/evidence/1",
"type":
"Evidence"
,
"narrative": "# Final Project Report \n This project was ...",
"name": "Final Project Report",
"description": "This is the final project report.",
"genre": "Research",
"audience": "Department"
},
{
"id": "https://github.com/somebody/project",
"type":
"Evidence"
,
"name": "Final Project Code",
"description": "This is the source code for the final project app.",
"genre": "Research",
"audience": "Department"
}
],
- https://1edtech.github.io/openbadges-specification/ob_v3p0.html#evidence
Example from OIDF eKYC and IDA Evidence
{
"evidence": [{
"type": ["https://openid.bitbucket.io/ekyc/openid-connect-4-identity-assurance.html#name-verification-element"],
"verification": {
"trust_framework": "uk_tfida",
"scheme": "idconnect",
"assurance_level": "idconnect_kyc_partial",
"assurance_process": {
"policy": "GPG_45",
"scheme_policy": "idconnect_kyc",
"procedure": "L1B",
"scheme_procedure": "dlnl_sm",
"gpg45_confidence_level": "low",
"assurance_details": [{
"assurance_type": "evidence_validation",
"assurance_classification": "3",
"evidence_ref": [{
"txn": "WED-85762937582385444",
"evidence_classification": "3"
}]
},
{
"assurance_type": "address_validation",
"assurance_classification": "2",
"evidence_ref": [{
"txn": "idv-7567445464543",
"evidence_classification": "3"
}]
},
{
"assurance_type": "verification",
"assurance_classification": "3",
"evidence_ref": [{
"txn": "BIO-8576293758238555"
}]
}
]
},
"time": "2021-05-11T14:29-01:00",
"verification_process": "7675D80F-57E0-AB14-9543-26B41FC22",
"evidence": [{
"type": "document",
"check_details": [{
"check_method": "vri",
"sub_check_method": "vri",
"organization": "wedocu",
"txn": "WED-85762937582385444"
},
{
"check_method": "bvr",
"sub_check_method": "bvr",
"organization": "biotrick",
"txn": "BIO-85762937582385555"
}
],
"time": "2021-04-09T14:12Z",
"document_details": {
"type": "driving_permit",
"document_number": "MORGA753116SM9IJ35",
"date_of_issuance": "2021-01-01",
"date_of_expiry": "2030-12-31",
"attachments": [{
"desc": "image_of_document",
"content_type": "image/png",
"content": "Wkd0bWFtVnlhWFI2Wlc0Mk16VER2RFUyY0RRMWFUbDBNelJ1TlRjd31dzdaM1pTQXJaWGRsTXpNZ2RETmxDZwo="
}],
"issuer": {
"name": "DVLA",
"country_code": "GBR"
}
},
"claims": {
"given_name": "Sarah",
"family_name": "Meredyth",
"birthdate": "1976-03-11",
"place_of_birth": {
"country": "UK"
},
"address": {
"locality": "Edinburgh",
"postal_code": "EH1 9GP",
"country": "UK",
"street_address": "122 Burns Crescent"
}
}
},
{
"type": "electronic_record",
"check_details": [{
"check_method": "data",
"sub_check_method": "cra_record_primary",
"organization": "theIdP",
"txn": "idv-7567445464543"
}],
"time": "2021-04-09T14:12Z",
"record": {
"type": "cra_account",
"authoritative_source": "equiexpunion",
"authoritative_source_reference": "gg54y4y5y5433443",
"created_at": "2021-01-01",
"date_of_expiry": "2030-12-31",
"date_last_updated": "2022-01-31",
"outcome_of_check": "positive",
"source": {
"name": "idvalidation4u",
"address": {
"street_address": "300 Bath Street",
"postal_code": "G2 4LH",
"country_code": "SCO"
}
}
},
"claims": {
"given_name": "Sarah",
"family_name": "Appleby",
"birthdate": "1976-03-11",
"address": {
"locality": "Leeds",
"postal_code": "LS12 6GT",
"country": "UK",
"street_address": "45 White Rose Road"
},
"start_date": "01-01-2020",
"end_date": "31-12-2020"
}
}
]
}
}]
}
Based on the 2 examples in the vc-specs dir, it seems people are using "type" in different ways...
if there are 2 independent implementations of either of these, I think evidence predicate can stay in the core spec, but we probably should add a note that it is interpreted based on type and possibly in very different ways.
The issue was discussed in a meeting on 2023-04-04
- no resolutions were taken
View the transcript
1.12. Improve tests for Evidence (issue vc-data-model#870)
See github issue vc-data-model#870.
Kristina Yasuda: By Orie, about evidence. Anyone willing to take this one?.
… To those who joined late, this call is about assigning issues that are not currently assigned. It means you are responsible that the issue gets addressed..
Oliver Terbu: +1 ivan.
Ivan Herman: I don't really understand.. The title says you have to improve tests, which suggests this is about tests, not the spec. The text talks about better defining the property or removing it. What is this about? Question is to Orie..
Orie Steele: I believe we had the spectre of objections in the context of our RDF terms for which there have been no implementation. I believe "evidence" is one of them. In order to retain this in the core spec, this was the original intention of the issue. As far as I am aware, we have not provided any concrete examples of using the "evidence" property..
… I think DavidC is the furthest toward a concrete example. If we have multiple concrete implementations, this is the testing we should have to retain the property..
Ivan Herman: I understand. I suggest we postpone this to when we get to CR exit criteria. We have to say what it means for a property to pass the CR line. The issue wasn't clear from the description..
David Chadwick: Part of the NGI Atlantic project, we did a simple example which referred to NIST and eIDAS levels of assurance. There would have been two independent implementations, but the project ran out of time..
Oliver Terbu: To proceed with this issue, does somebody have to register an "evidence" type in the directory, and then write a test that uses this "evidence" type? Implementers don't need to anything, but they should not fail if they encounter an "evidence" property of this specific type, is this correct?.
David Chadwick: This is a discussion we had with termsOfUse and nonTransferable, whether these properties can be ignored by recipients. I think DavidC added something to this discussion..
Oliver Terbu: Whoever gets assigned to this issue, should be someone who knows an "evidence" type that is used somewhere, and it should be registered in the VC directory. Does this make sense?.
Dave Longley: Trying to respond to ignoring extension points. This was certainly the intention..
Kerri Lemoie: We have OpenBadges 2.0 use "evidence". I will share a link..
Kerri Lemoie: See Open badges reference.
Kerri Lemoie: It's a bit confusing, because they are using the "evidence" property from the VC spec, but they are using it in their own way with their own properties. This is important for EDU community, but they have their own testing. There is overlap, not sure what to do with it. I don't know if we can test it if they already test it in their own way..
Dave Longley: +1 to you must be able to ignore unrecognized terms.
Michael Jones: It's not a question whether they can ignore "evidence". They must be able to ignore it. In our test suite we should inject random elements and ensure that implementations don't fail if they encounter them..
Kristina Yasuda: Anyone willing to be assigned to this issue?.
Oliver Terbu: You can assign me to find out what actually needs to be done, if anything needs to be changed in the test suite..
Kerri Lemoie: oliver - feel free to reach out if you want insights into Open Badges.
@OR13 It is obvious that the Open Badges example is wrong and the OIDF example is correct, given the standard definition "Each evidence scheme is identified by its type. The id property is optional" The type must be a URI.
@OR13 To me it is not entirely clear what this issue is about. The title of the issue seems to imply we have to update the test suite. In that case, we should move this issue to https://github.com/w3c/vc-test-suite. Would you agree? If this is not the case, then we should close this issue and create a new one that describes what you wanted to get fixed. If you want to remove evidence entirely from the W3C VCDM core spec because the property is lacking two independent implementations of a type registered in the VC directory, then we should create a separate issue for that.
For reference, the W3C test suite has one test case for evidence, which uses the following evidence object:
"evidence": [{
"id": "https://example.edu/evidence/f2aeec97-fc0d-42bf-8ca7-0548192d4231",
"type": ["DocumentVerification"],
"verifier": "https://example.edu/issuers/14",
"evidenceDocument": "DriversLicense",
"subjectPresence": "Physical",
"documentPresence": "Physical"
It seems that the type DocumentVerification has two problems at least:
- not included in
@context - not included in VC directory
I'm supportive on using an evidence type in the W3C test suite that has been registered in the VC directory instead. Currently, there is just one which is the OIDF eKYC one which doesn't define a context/vocab which makes it hard to be included in the W3C VC test suite.
There should be 2 places where tests are improved:
- tests for the core data model
- tests for securing formats that use evidence
I'm not volunteering to do either for evidence, but if neither is done well, I will argue against continuing to keep evidence in the core spec.
the type
DocumentVerificationhas two problems at least
A third is that this type is not a URI.
I believe this is blocked by w3c/vc-data-model#1082 since if we decide to include evidence in the reserved property list, we will have to remove the test cases that contain evidence. On the other hand, if we decide to keep evidence, we need to update the test cases to be fixed like @OR13 described it above, 1) in the core data model, 2) in the securing formats.
@OR13 let's close this issue after w3c/vc-data-model#1082 was resolved. If tests for evidence still have to be improved, please close this issue and open a new issue in the test suite repository.
Could @awoie or @OR13 please state what you expect the conformance tests to test. It seems to me that we could have tests along the following lines
- send a correctly formatted Evidence object in a correctly formatted VC to the system under test and expect it accept the VC (Note. the system under test does not need to process nor understand the Evidence object. There is no requirement to do this).
- send an incorrectly formatted Evidence object with the type parameter missing to the system under test and expect it to reject the VC I think these two tests are the minimum that are needed, but may be sufficient for the test suite
Ideally we could test both conformance to the data model, and business logic associated with evidence production and consumption.
The latter would need to be in a different test suite from the core data model.
I agree that testing business logic should not be part of the VCDM test suite. My question to you was about tests for the VCDM.
The issue was discussed in a meeting on 2023-07-05
- no resolutions were taken
View the transcript
2.11. Evidence extension point (was: Improve tests for Evidence) (issue vc-data-model#870)
See github issue vc-data-model#870.
Kristina Yasuda: Is this addressed by evidence being in the reserved properties, Orie?
Orie Steele: Sort of.
… It's pretty bad form to reserve something that you can't find any interoperable implementations of.
… If there's a stronger evidence type, that would be better to revert to.
Kristina Yasuda: I think that EBSI and Kerry in EDU are using evidence.
Orie Steele: right, but each party using
evidenceis using thepredicateand AFAIK, thetypehas no interop?
Phillip Long: In the education community, it's often used with assertions of skills and confidences.
Greg Bernstein: Phil is evidence in the new clr 2 spec?
Phillip Long:
@GregB- lemme look.
Greg Bernstein: CLR 2.0 https://www.imsglobal.org/spec/clr/v2p0#evidence.
Phillip Long:
@GregBit's used frequently in the CLRv2: see https://www.imsglobal.org/node/205106#evidence, and, https://www.imsglobal.org/node/205106#evidence-0 for examples.
Manu Sporny: I see that Gabe is saying that TBD will be using it.
… The OpenID eKYC-IDA spec uses it.
Orie Steele: fwiw, i dont think anyone is using it in any interoperable way... and 2, we have no ability to encourage interop based on what we have been doing.
Manu Sporny: We need to see some level of interop demonstration.
… I believe you need at least 2 interoperable implementations to reserve a term.
… There are a number of people using it in some capacity.
… But interop has yet to be demonstrated.
David Chadwick: You can test that 2 implementations will work with the evidence property.
… Testing the business logic is another thing altogether.
… I don't think our spec specifies the business logic.
Manu Sporny: That was the model we were working under in the V1 and V1.1 WGs.
… It seems like the V2 WG is upping the bar.
… It seems to be what people are pushing towards.
Orie Steele: +1 Manu.
Manu Sporny: Several accepted PRs have asked for more.
Kristina Yasuda: What does that mean in terms of the spec text or directory?
David Chadwick: I don't see how you can put those conformance tests into the VCDM test suite.
… There are different applications that will have different specifications of evidence.
… They will be specific to those groups of users and won't be generic.
Orie Steele: If the VCDM defines the evidence property and it's defined in the normative @context then there are positive and negative data model tests than can be applied to it.
… even if what the property means is completely untestable.
David Chadwick: +1 to Orie.
Kristina Yasuda: I'm leaving this open. The IRC comments will be reflected in the issue.
Orie Steele: can we cite IMS global normatively?
Phillip Long: @Orie - yes, they have six implementers as I recall.
Orie Steele: maybe good to do that, when keeping the property.
Manu Sporny: The next step for this issue should be to refer to Phil's citations above.
Phillip Long: I can be assigned this issue. I will add citations to the issue.
Kristina Yasuda: Brent will chair next week.
… We will probably cancel in two weeks.
There is a strong reliance in the education and training implementations of the VCDM for the evidence property. In the OBv3 specification.
Example from 1Edtech Open Badges v3.0 specification in Final Candidate Public release
"evidence": [ { "id": "https://1edtech.edu/credentials/3732/evidence/1", "type": "Evidence" , "narrative": "# Final Project Report \n This project was ...", "name": "Final Project Report", "description": "This is the final project report.", "genre": "Research", "audience": "Department" }, { "id": "https://github.com/somebody/project", "type": "Evidence" , "name": "Final Project Code", "description": "This is the source code for the final project app.", "genre": "Research", "audience": "Department" } ],* https://1edtech.github.io/openbadges-specification/ob_v3p0.html#evidence
It is also used in the 1Edtech CLRv2 (compound credential including a transcript, OBv3s and CASE competency statements).
There are commitments to implement these VC compatible 1Edtect credentials by numerous platform providers including: Territorium, AwardAssured, & Randa Solutions. Several others have expressed intent but aren't far enough along, or at least willing to say publicly their timeline.
The issue was discussed in a meeting on 2023-07-26
- no resolutions were taken
View the transcript
2.1. Evidence extension point (was: Improve tests for Evidence) (issue vc-data-model#870)
See github issue vc-data-model#870.
Brent Zundel: Issue 870. Should we do this before or after CR and who to assign?
Manu Sporny: https://w3c.github.io/vc-specs-dir/#evidence.
Manu Sporny: https://www.imsglobal.org/spec/ob/v3p0/.
Manu Sporny: https://1edtech.github.io/openbadges-specification/ob_v3p0.html#evidence.
Manu Sporny: I think we need to resolve it before CR. It's missing 1EdTech - they very clearly use this extension point.
Reviewing the thread above it seems there are a couple of different issues here. One is the question of testing the evidence extension point. A second is about the use of the Evidence, either to include corroborating information about the claim being made in the credential or as a pointer to such corroborating information and how that should be represented. Lastly there is a question about aligning the term Evidence with its definition by NIST, which is specific to evidence concerning ensuring the security of a system, which is very specific to a that particular use case.
Two immediate suggestions:
First David Chandwick notes that a test could be made to assure the Evidence object is correctly formatted or not. The 'test" would be to send a correctly formatted and the incorrectly formatted credential with the Evidence object in it. The first should pass and the second should fail. This makes sense and I think creating a PR to this effect in vc-test-suite.
Finally there is an issue with the proper syntax of the Evidence Object. Its use by OIDF eKYC and IDA Evidence relative to the use of Evidence by 1Edtech in the OBv3 specification.
My preference is to stay consistent with the VCDM v1.1
This specification defines the evidence property for expressing evidence information.
evidence The value of the evidence property MUST be one or more evidence schemes providing enough information for a verifier to determine whether the evidence gathered by the issuer meets its confidence requirements for relying on the credential. Each evidence scheme is identified by its type. The id property is optional, but if present, SHOULD contain a URL that points to where more information about this instance of evidence can be found. The precise content of each evidence scheme is determined by the specific evidence type definition.
Id there is a difference between how 1Etech defines evidence this can be raised in their credentials workgroup and a PR entered into their Github repo to address changing this.
What is important and clear is the need for Evidence and its importance to substantiate claims being made in a credential.
My concern and issue with the current definition of the evidence property shown above is that it is only applicable to an entire VC.
I believe it should be applicable to any assertion/claim (or group of same) contained in a VC — in other words, that the issuer may say "I relied on this evidence for these claims, and this other evidence for these other claims" and the verifier may decide that "these claims" may be (sufficiently) relied upon, but "these other claims" may not be relied upon at all.
TallTed - I'm assuming you're saying the vcdm v1.1 prevents use of the element property to make multiple assertions. Can you elaborate why you come to that conclusion?
It's not that the VCDMv1.1 prevents use of the evidence property to make multiple assertions. It's that the definition of the evidence property clearly scopes (each value of) that property to apply to the entire credential, not to any partial set of claims or assertions taken from within the credential. Again —
The value of the evidence property MUST be one or more evidence schemes providing enough information for a verifier to determine whether the evidence gathered by the issuer meets its confidence requirements for relying on the credential.