api Consider a `provides` property on Annotation

... for when the annotation provides some feature or functionality to the target resource(s), either directly or by using the body resource(s).

For example, a client would benefit from knowing that an annotation provides captions to an AV canvas (or multi-media scene) via the body VTT resource.

This would be parallel to the accessibility property already available on body and target resources, which conveys that the resource has the particular feature already. For example, a video with burned in captions has the accessibility feature of openCaptions.

The initial list of features for provides would be from the a11y vocab list, to mirror the existing property: https://www.w3.org/community/reports/a11y-discov-vocab/CG-FINAL-vocabulary-20230718/#accessibilityFeature-vocabulary

However in the future we could add our own entries to cover new use cases.

An example annotation:

{
  "type": "Annotation",
  "provides": ["closedCaptions"],
  "target": "https://example.org/iiif/canvas/1",
  "body": "https://example.org/resources/video1.vtt"
}

( /cc @nfreire @glenrobson)

Jul 03 '24 18:07 azaroth42

I think this would be a good solution for the use cases identified by the AV Annotations Motivations TSG. The schema.org vocabulary covers the use cases related with open captions, audio descriptions and transcriptions. The only case that is missing is subtitles.

Comparing with the alternative solution of using the "behavior" property with values "overlay" and "sidebar", this proposal makes it more explicit what kind of functionality is provided by the annotations.

Jul 05 '24 08:07 nfreire

This was discussed in the AV Community Call on 9/9/24. Overall, lots of positive feedback on this suggestion and much preferred over a behavior property with values overlay and sidebar.

Some questions came up:

Will the available vocabulary include everything in the accessibilityFeature property (section 4.2) or just a defined set that matches the AV Annotations TSG use cases? There is support and interest in allowing broader use of this vocabulary, but also concern that 4.2.1 (Structure and Navigation Terms) could cause confusion with how manifest producers are supposed to implement things like table of contents.
Would this property be available for non-AV usage? Initially interest in wondering how things like 'transcripts' might be applied to manuscripts, etc. for accessibility purposes.
There may be a need to add new vocabulary as we continue to use AI to develop new kinds of structured annotations. Presumably this could be done within IIIF itself, rather than with schema.org? We already have a potential new vocabulary option: subtitles

Sep 09 '24 19:09 elynema

A non-AV use case came to my mind after the discussion in the AV Community Call on 9/9/24. It is the case of supplementing annotations on a canvas (with an image), where the annotation contain the text from OCR with position information on the image. These annotations are not meant to be shown to the user, they are intended be used to highlight words on the image after a user search via IIIF Content Search. A value for the provides property to inform the IIIF client that such annotation are meant to provide this functionality could also be considered.

Sep 10 '24 11:09 nfreire

Discussed on Editors call 2024-09-25

Will the available vocabulary include everything in the accessibilityFeature property (section 4.2)

@elynema You identify the problem - there are terms in that vocabulary that we definitely should not be using (e.g., don't use the "annotations" term as a value of provides, we already have the .annotations property!)

And there are terms that we maybe could use but the value (and the downsides) aren't clear.

Would this property be available for non-AV usage?

I think this is related and helps focus any process of going through that list of terms and marking some that we should consider as recommended values of provides. There is considerable overlap with the needs of accessibility in general in IIIF (cc @brittnylapierre). It seems like we should focus on accessibility, convene a session or two of interested parties and go through the list to rule some in and rule some out. For each term we would need to establish what the expected behavior is - what does a IIIF client do with the information? And obviously suggestions can be added (perhaps to this issue) independently of such a meeting.

There may be a need to add new vocabulary...

New terms can be first tried out with experiments and shared practice, then become more formalized as a published IIIF extension, and eventually, if appropriate, make their way into core spec (e.g., navPlace could be part of Presentation 4). And IIIF can provide its own identity for such terms if nothing fits from existing external vocabularies.

Oct 02 '24 10:10 tomcrane

A provides property would definitely be useful for print materials too, and I think will facilitate a lot of the discussions on how we can leverage the accessibility guidelines for EPUB and PDF documents in some new analogous IIIF specifications. You can see an initial brain storm here - https://groups.google.com/g/iiif-discuss/c/lj5YaBo_l1s

The 'accessibility metadata' portion ended up with us thinking that the presentation API spec will need a property where we can add information for clients on rendering the specified accessibility feature, which is exactly what you are all talking about!

The major accessibility features that annotations could be involved in for IIIF representations of print materials are:

Providing text alternatives to images within the image file ( See analogous EPUB guidelines here )
Providing structural navigation ( See analogous PDF guidelines here ) note: would need a provides structuralNav with HTML annotations
Markup for tables within images ( See analogous PDF guidelines here )

We also think Ranges will be useful for providing bookmarks/table of contents functionality, which is also an accessibility feature for both EPUB and PDF documents, see here and here respectively.

I agree with your proposed approach @tomcrane and am excited to get us all together to hash out the details. I also like the idea of creating an extension at first as well!

Oct 03 '24 01:10 brittnylapierre

Today at work, we discussed how more about how annotations can be used to provide text alternatives for images or videos in IIIF.

Many institutions use AI to generate this text on a large scale, often without human verification. As a result, some of the text may be inaccurate, similar to YouTube's auto-generated captions.

Given that these text representations are often auto-generated, we might need a way to flag each annotation to indicate whether the text has been verified or corrected using annotation editing software.

Maybe something like "autoGenerated" could be explored. This can help people relying on the text content to have an indicator of how reliable it will be.

Edit: I realize now we can leverage HTML in annotations for structural nav in print materials!

{
  "id": "https://iiif.io/api/cookbook/recipe/0021-tagging/annotation/p0002-tag",
  "type": "Annotation",
  "motivation": "supplementing",
  "provides": [ "structuralNavigation" ],
  "autoGenerated": true,
  "body": {
     "type": "TextualBody",
     "language": "de",
     "format": "text/html",
     "value": "<h1>Header text goes here</h1>"
  },
  "target": "https://iiif.io/api/cookbook/recipe/0021-tagging/canvas/p1#xywh=265,661,1260,1239"
}

or, for video content:

{
  "type": "Annotation",
  "provides": ["closedCaptions"],
  "autoGenerated": true,
  "target": "https://example.org/iiif/canvas/1",
  "body": "https://example.org/resources/video1.vtt"
}

Oct 07 '24 19:10 brittnylapierre

Isn't this just purpose? https://www.w3.org/TR/annotation-model/#motivation-and-purpose

Oct 10 '24 15:10 dnoneill

Thanks @dnoneill !!

It looks like purposes has the following values provided under the motivation property: https://www.w3.org/TR/annotation-model/#motivation-and-purpose (assessing to tagging) Based on my reading of: https://www.w3.org/TR/annotation-vocab/#haspurpose

Then the IIIF presentation API spec says:

Additional motivations may be added to the Annotation to further clarify the intent, drawn from extensions or other sources.

So then, if I understand correctly, it looks like we might be able to create an extension to extend "purpose", with additional values for accessibility-related purposes right @dnoneill @tomcrane @glenrobson ?

For example, for print materials:

{
  "id": "https://iiif.io/api/cookbook/recipe/0021-tagging/annotation/p0002-tag",
  "type": "Annotation",
  "motivation": "supplementing",
  "autoGenerated": true,
  "body": {
     "type": "TextualBody",
     "language": "de",
     "format": "text/html",
     "value": "<h1>Header text goes here</h1>",
     "purpose": "structuralNavigation"
  },
  "target": "https://iiif.io/api/cookbook/recipe/0021-tagging/canvas/p1#xywh=265,661,1260,1239"
}

or, for video content:

{
  "type": "Annotation",
  "autoGenerated": true,
  "target": "https://example.org/iiif/canvas/1",
  "body": {
    "source":  "https://example.org/resources/video1.vtt",
    "purpose": "closedCaptions",
  }
}

Or did we want to create an extension with a property of with the name of 'accessibilityFeatures' (might be better than 'provides') which draws on the accessibilityFeatures values

For example, for print materials:

{
  "id": "https://iiif.io/api/cookbook/recipe/0021-tagging/annotation/p0002-tag",
  "type": "Annotation",
  "motivation": "supplementing",
  "accessibilityFeatures": ["structuralNavigation"],
  "autoGenerated": true,
  "body": {
     "type": "TextualBody",
     "language": "de",
     "format": "text/html",
     "value": "<h1>Header text goes here</h1>"
  },
  "target": "https://iiif.io/api/cookbook/recipe/0021-tagging/canvas/p1#xywh=265,661,1260,1239"
}

or, for video content:

{
  "type": "Annotation",
  "accessibilityFeatures": ["closedCaptions"],
  "autoGenerated": true,
  "target": "https://example.org/iiif/canvas/1",
  "body": "https://example.org/resources/video1.vtt"
}

I'm not sure what would be the best path to take personally! My only thought is if we extend purpose would we be able to outline to clients how to render the annotation for better accessibility?

Oct 10 '24 16:10 brittnylapierre

purpose is on SpecificResource not generally on resources used as body or target. See: https://www.w3.org/TR/annotation-model/#purpose-for-external-web-resources

Basically purpose is just motivation for specific resources.

It's also different from providing an accessibility feature. The motivation could be describing but part of the result of that description is increased accessibility.

accessibilityFeatures would limit future reuse with non accessibility related provisions. This issue isn't the issue to discuss all things accessibility related, just provides. For example, https://github.com/IIIF/api/issues/1288 is about altText.

Oct 10 '24 17:10 azaroth42

@azaroth42 Oh do extensions usually only map to one property? If so, I might need to create a new issue here for concerns I'm hearing about auto generated content, unless I can find an existing one.

Do you think that print materials could suffice with: https://www.w3.org/TR/annotation-model/#accessibility-of-content that you pointed out in your original post, ex:

{
  "id": "https://iiif.io/api/cookbook/recipe/0021-tagging/annotation/p0002-tag",
  "type": "Annotation",
  "motivation": "supplementing",
  "body": {
     "type": "TextualBody",
     "language": "en",
     "format": "text/html",
     "value": "<h1>Header text goes here</h1>",
     "accessibility": "structuralNavigation"
  },
  "target": "https://iiif.io/api/cookbook/recipe/0021-tagging/canvas/p1#xywh=265,661,1260,1239"
}

(It looks like this might be the solution I'm looking for with my use case!)

Then your 'provides' would be targeting annotations with a non-object body, like in your original example?

Oct 10 '24 17:10 brittnylapierre

Extensions can do many things all at once, but if we want a chance to get something into 4.0, it should follow the design principles for IIIF, and be specific as to which use cases it enables and which it doesn't. If there are reasons to not use provides for the purpose laid out in the issue then that would be great to discuss here. For general accessibility issues, I think a new issue would be easier to track -- for example, this issue would be closed when either provides is added or it is determined not to be appropriate, and we don't want to lose the more general discussion.

Oct 10 '24 17:10 azaroth42

Okay! Thank you - I will create a separate issue for discussing the auto generated content

Oct 10 '24 17:10 brittnylapierre

The AV Annotations TSG has gathered some feedback on the existing accessibilityFeature vocabulary, and its usefulness / appropriateness for use in IIIF. Hopefully this will be helpful in drafting the updates for the provides property in the IIIF spec. See https://docs.google.com/document/d/1lrvzfBgrSiYa-hlRxGd8EYi52ZD0xX5myR5G4P58eOk/edit?usp=sharing.

In addition, the AV Annotations TSG is working on draft updates to existing recipes and new recipes that will demonstrate how we would use this new property:

Updated recipes

Update recipe 219 to use provides property for captions: https://github.com/IIIF/cookbook-recipes/issues/555
Update recipe 74 to use provides property for both captions and subtitles in multiple languages: https://github.com/IIIF/cookbook-recipes/issues/556

New recipes

Create recipe 79 for timed text annotations for captions that use provides property: https://github.com/IIIF/cookbook-recipes/issues/79
Create recipe 253 for transcript that links to an external file using provides property: https://github.com/IIIF/cookbook-recipes/issues/253
Create recipe 554 for audio transcription for video content using provides property: https://github.com/IIIF/cookbook-recipes/issues/554

Jan 14 '25 16:01 elynema

The Annotations TSG have worked on the following recipes to see if provides matches the use cases. Here are the latest examples:

Using Caption and Subtitle Files in Multiple Languages with Video Content an example with two vtt files with different provides properties
Using Annotations for Timed Text an example which includes supplementing annotations and the provides is added to each annotation. Note this isn't ideal as the viewer has to look through every annotation to see if it should go into the captions part of the viewer.
Using Transcript Files with Audio or Video Content single annotation pointing to a vtt with a provides value of transcript.
Using Caption Files with Video Content single annotation pointing to a vtt which provides closed captions

Note these recipes will be taken to the TRC post publishing of v4 and I am going to prototype a v4 cookbook and we will re-point these recpies to the v4 cookbook branch.

Mar 18 '25 20:03 glenrobson