[Proof of Concept] Video Export using Web Codec API

[lucasdinonolte]

This PR explores a way to use the Web Codec API to export video. This is not only faster than using the JS implementation we’re currently using, but also opens up more export formats (including H.264 codec in MP4 container).

This is in the early stages of exploring. Video formats, codecs and container are a fairly complex topic and I feel like I'm just starting to understand.

What this does

• Adds VideoWriter class to mechanic core, replacing the previous WebmWriter
  • uses Web Codec API to export video to MP4 (using H.264 Codec) or WebM (using VP9 Codec)
  • falls back to the pure JS WebmWriter for browsers that don’t support web codec API
  • errors for non-supported browsers if a user wants to export MP4 (as their currently is no JS fallback for the H.264 codec)
• Allows animated design function to specify their animationFormat to set the desired export
  • allowed values webm and mp4 (defaults to webm, as we have a fallback for non-supported browsers)
• Allows animated design functions to specify their desired animationBitrate for export
  • If non is set (default) mechanic will calculate a high quality bitrate from width, height and framerate

What could be done

• Add video export presets
  • Right now it only provides the options mp4 and webm
  • ... but this could easily be extended to provide presets like "High Quality MP4", "Small File Size Mp4", etc.
• Discuss fallbacks for non-supported browsers
  • Web Codec API is still a working draft and browser support outside of Chrome is not looking too good
  • For webm video we can fallback to the WebmWriter that was used previously
  • For mp4 video we could investigate some of the WASM based H.264 encoders
• Discuss if we want the export format as an option on the design function or if we should let the end user choose the export format

Test Driving

• fillAnimatedCanvas is set up to export MP4 video

There’s probably a million more things to discuss that I can't even begin to think of, as video-formats, containers, and codecs are quite the rabbit hole. I tried to leave extensive code comments to document what is going on and why certain decisions were taken. But please challenge everything I've done :blush:

Caveats

Not all browsers come with full support for the web codec API, Firefox for example has no support for the Web Codec API currently at all. But there's also differences in Chromium based browsers: Chromium (as it's used on Linux or in headless scenarios) only comes with built-in support for open codecs and lacks proprietary codecs. This means no h.264 in Chromium. Google Chrome however has full support including proprietary codecs.

However, when using Chrome, the licensing of h.264 is covered by Chrome's license and we’re good to use it.

Alternatives

I did a test drive using ffmpeg-wasm locally and it works across browsers (including headless Chromium), so this could be a feasible alternative to relying on the Web Codec API. However ffmpeg-wasm comes at the cost of downloading ~25MB of WebAssembly code to the user's browser first.

As h.264 is a proprietary codec there is some confusion around whether we'd be allowed to embed it (in the form of ffmpeg-wasm). There is some indication that we might don’t want to do this (or that we could do it, but need to inform our users about this, so they don’t get into trouble).

For references see:

• https://ffmpeg.org/legal.html
• https://github.com/mattdesl/mp4-h264#project-suspended
• https://www.via-la.com/licensing-2/avc-h-264/avc-h-264-license-fees/

[netlify[bot]]

✅ Deploy Preview for dsi-logo-maker ready!

Name	Link
🔨 Latest commit	084be955b4062141544f38970ad1143bb7678d9b
🔍 Latest deploy log	https://app.netlify.com/sites/dsi-logo-maker/deploys/647f6fa7ab337f00084cae81
😎 Deploy Preview	https://deploy-preview-178--dsi-logo-maker.netlify.app
📱 Preview on mobile	Toggle QR Code... Use your smartphone camera to open QR code link.

---

To edit notification comments on pull requests, go to your Netlify site settings.