Add support data for image formats/codecs? (especially WebP) And media (video/audio) formats/codecs too?

[sideshowbarker]

As far as I can see we’re not yet storing support data in BCD for image formats/codecs.

The level of browser support for most image formats is the same across all browsers, so in general the data is arguably not something that web developers have a strong need for in practice.

But WebP at least seems like an exception to that; Apple only recently announced that they’d added support for it, in Safari 14 — but per https://developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types#WebP_image and https://wiki.developer.mozilla.org/en-US/docs/Web/Media/Formats/Image_types$compare?to=1647119&from=1643708 it seems that it’s not actually supported in desktop/macOS Safari yet but instead only on Mobile/iOS Safari.

https://caniuse.com/webp does have support data for WebP. So that’s further evidence that MDN probably should too.

And while the level of browser support for image formats in general may be the same across all browser, the level of browser support for video and audio formats and codecs is not — it’s much more complicated.

https://caniuse.com/?search=video%20format shows some relevant data.

Difficulties in being able to deliver video content in a format/codec that works across browser has been a pretty well-known pain point for developers — so having related support data for video and audio formats and codecs in BCD seems even more important and useful to developers than having it for image formats.

[saschanaz]

(This might also be helpful for AVIF support.)

[Elchi3]

the level of browser support for video and audio formats and codecs is not — it’s much more complicated.

Yes, this. We failed to migrate formats/codecs compat data from the wiki into BCD because of its complexity. How to best structure the data has been a previous discussions on this matter, too:

• https://discourse.mozilla.org/t/deciding-on-how-to-track-media-format-information/35531
• https://github.com/mdn/browser-compat-data/pull/4193

[ddbeck]

@Elchi3 linked to the prior issue and discussion of note on this. To summarize those links a little bit, some of the major sticking points were:

• Dealing with the huge number of compat entries to completely describe the availability of different container/codec combinations (i.e., you'd have at least one support statements for container × codec × browser)
• Substantiating claims (e.g., Chrome depends on FFmpeg, so we might have to dig through FFmpeg's history to understand when something landed in Chrome)

I think we need to have a good idea of how to navigate those issues (without driving ourselves mad), if we're going to include them in BCD.

[dontcallmedom]

* Substantiating claims (e.g., Chrome depends on FFmpeg, so we might have to dig through FFmpeg's history to understand when something landed in Chrome)

FWIW, esp for evergreen browsers, having data on what's currently supported is already very valuable - I wouldn't make historical support data an obstacle to surfacing that value.

[hamishwillee]

IMO this is absolutely worth doing. It is not substantively different than the early days of web API data and the current status of HTTP - just knowing that something is supported in some way or definitely not supported is incredibly useful. Over time that story can be extended.

Doing this in BCD would provide structure and process that can't really be managed keeping this data in the docs. And clearly hasn't been. At least in BCD we'll try to share the load of keeping it updated.

Given the complicated matrix of support it is only possible if we constrain the scope and set expectation on what the data means.

Looking at the sticking points in https://github.com/mdn/browser-compat-data/issues/6971#issuecomment-729723722 :

• Dealing with the huge number of compat entries to completely describe the availability of different container/codec combinations (i.e., you'd have at least one support statements for container × codec × browser)
• Substantiating claims (e.g., Chrome depends on FFmpeg, so we might have to dig through FFmpeg's history to understand when something landed in Chrome)

The problem is bigger than this if we were to try and do this completely methodically. It isn't just container > codec > browser, it's element > "purpose" > container > codec > browser. To expand on that by example, some container/codec options that are supported for

• video are not supported for audio (in whatever elements display those things)
• elements are not supported for WebRTC and visa versa
• displaying in canvas are not supported for exporting a canvas as an image

Doing this for all the combinations would be unmaintainable, but IMO we can still do a useful and valuable job by constraining the scope.

Specifically:

1. Limit ourself to the more important container and codec combinations. We do not have to cover every single case or combination.
2. Start with the things we have data for - and feel free to use true and null or >= to allow us to fill this out over time.
3. Set expectation that the data indicates "some level of support" in the browser.
   • So if we're saying that FF supports WebP and VP8 video codec then it is supported for some cases, but not necessarily every possible case.
   • We can use notes to clarify specific uses if we know them - but we don't get hung up on that. We could expand out with subfeatures later on if we want for the specific use cases.

My thinking is that we use the obvious simple structure:

• media (folder)
  • webp.json (container)
    • subfeatures are supported codecs
      • notes used for specific container/codec use.
  • wav.json (container)

If it can be done "easily" it would be good to further be able to filter for display based on whether a codec is audio, video. That would allows us to show, say, just the audio codecs with elements that can only play audio.

Thoughts? Would be great if we could at least prototype.

[caugner]

My thinking is that we use the obvious simple structure:

* media (folder)
  
  * webp.json (container)
    
    * subfeatures are supported codecs
      
      * notes used for specific container/codec use.
  * wav.json (container)

Late to the party, but couldn't we have a grouped structure, with intermediate levels:

// media.codecs.audio.json
media.codecs.audio.aac
media.codecs.audio.mp3

// media.codecs.video.json
media.codecs.video.h264
media.codecs.video.vp8
media.codecs.video.vp9

// media.containers.json
media.containers.mp4
media.containers.mkv
media.containers.webm

// media.format.image.json
media.format.image.jpeg
media.format.image.webp

Note that media.codecs/containers/format, media.codecs.{audio,video}, and media.format.image wouldn't need support statements themselves.

[gsnedders]

This will also heavily run into https://github.com/mdn/browser-compat-data/blob/ccd9d7fe743bcb87e7400ec57022c2dd7de743f2/docs/data-guidelines/index.md#operating-system-limitations-imply-partial_implementation (and also hardware limitations, given e.g. HEVC support in Chrome apparently depends on hardware support on Windows, Linux, and ChromeOS).

Given we'll have such a large percentage of any table being "partial support", I almost wonder if it's worthwhile having OS-specific subfeatures, though that would be a break from current practice.

[bgrins]

Would it be simpler to start with images? Browser support should be more consistent across platforms with software decoding.

[caugner]

Would it be simpler to start with images?

Yes, absolutely.

I will bring it up in the next BCD meeting, and we should be able to use this image format test to determine first supported versions for all the formats mentioned on MDN.

[hamishwillee]

The other thing to consider is "support for images/video in what context". Webp is pretty widely supported, but not necessarily in something like manifests, since that depends both on browser and the OS image support in the GUI. Similarly some formats of video are supported in the web pages, but not in WebRTC and visa versa.

I still hold it would be good to show broad support, but I think this will be hard to be canonical on.

[caugner]

I have now:

• Extended the image format test to include a column for background-image, and added a separate noscript test.
• Opened https://github.com/mdn/browser-compat-data/pull/27168 to add initial data for all image formats.

Please have a look at the PR and give feedback there if you can. 🙏

[chrisdavidmills]

In https://github.com/mdn/browser-compat-data/pull/27142, I've tried to add support for new codecs in the MediaRecorder API by adding data points for including those codecs in the MIMEType codec string. This is how it works in MediaRecorder — you specify the MIME type you want to record the media in.

[bsmth]

See also these issues requesting related content:

• [ ] https://github.com/mdn/content/issues/20570
• [ ] https://github.com/mdn/content/issues/33781