Add H.264 decoding by utilizing OpenH264 from Cisco

[torokati44]

~~This is very much a WIP.~~ Progresses #723. ~~Basically pushing only for visibility.~~ Will partially replace #12539 (first only on desktop, in the future, using WebCodecs, entirely).

AFAIK this is the only legal way (without us or our users having to pay royalties to ~~MPEG LA~~ VIA LA) to do this with reasonable portability on desktop, but IANAL. See this little explainer video: https://vimeo.com/79578794

The code is somewhat based on the openh264-rs crate, see https://github.com/ralfbiedert/openh264-rs/issues/43.

YUV->RGB colorspace conversion is done by our trusty handcrafted code, like for H.263 and VP6[A]. This means that it will always assume BT.601 coefficients, even though it could theoretically, possibly be BT.709 or something like that. But the difference is probably subtle enough anyway.

TODOs, in no particular order:

• [x] Add a configure_decoder (or similar) method to the video backend/decoder traits. ~~Currently the preload method is misused to pass the "extradata" to the decoder.~~
  • Done.
• [x] Figure out how to not create colliding Indexes for proxied and handled videos in the ExternalBackend.
  • Added an enum to store in the new backend's arena. TODO: Add a warning comment.
• [x] Optimize copying YUV data out, and possibly AVCC->AnnexB format conversion
• [x] Handle downloading and hash-checking of the decoder shared libraries for the current platform/arch.
  • Done. TODO: Needs to do error handling better for proper fallback and hints for the user.
• [x] Add the Cisco license text, display it somewhere in the player.
  • Done. The new backend will write it into the current working directory as OpenH264-license.txt upon creation.
• [x] Add a command line option to disable the decoder (as per Cisco's license terms, I hope this will be enough)
  • Done: --enable-openh264 [true|false]
• [x] Switch to preferences for the enablement toggle
• [x] Figure out how, or if, to test this (command line flag for implicit agreement to fetch the decoder library on every run?)
  • Download on first run without confirmation, even in CI.
  • For tests, this should be opt-in, just like "video" and "renderer" in general.
  • Command line flag will be only for opting out.
• [x] Write up a method for updating OpenH264 versions in the future.
• [ ] Check if undoing the "start code emulation prevention" for the decoder config (extradata) is needed, and undo it if so.
  • Left this as a TODO comment for now...
• [x] Look at the presentation time offsets as provided by the FLV demuxer (later the MP4 container), and the fields in the SBufferInfo struct from the decoder.
• [x] General code cleanup, including properly working features to disable this, comments, and error handling.
  • Almost done!
• [x] Testing, testing, testing...
  • There is a visual test added, which is something...

TODOs for later:

• WebCodecs implementation for web. Consider whether MSE plus a hidden <video> element would be a feasible fallback - probably not, but who knows. I plan to put this into the same ExternalVideoBackend added here. Also with the "other ideas" below. Spares us from duplicating the proxying logic. The Android backend will have to be different though, as it's somewhere else.

Somewhat less closely related TODOs for later:

• MP4 (F4V) support: https://github.com/ruffle-rs/ruffle/pull/14655
• AAC-LC decoding support (already available in Symphonia).

Other ideas:

• Different, specific implementations for different platforms: Video Toolbox on macOS, Microsoft Media Foundation or DXVA[2] on Windows, VA-API (or VDPAU...) on Linux, and MediaCodec on Android. Maybe Vulkan Video anywhere applicable. (Loading FFmpeg or GStreamer on Linux would be iffy due to uncertain licensing of the used decoder.)

[torokati44]

This is what I use to test this at the moment: f4vplayer_h264_flv.zip

I hope it's okay to upload here. Despite the name, it now loads an FLV file. It's originally from here: https://permadi.com/2010/02/flash-playing-flvf4v-videos-with-script/

~~Of course, the decoder library still has to be obtained manually at the moment.~~

[danielhjacobs]

Will partially replace #12539 (first only on desktop, in the future, using WebCodecs, entirely).

The VideoDecoder part of WebCodecs (and many other APIs) are only available in secure contexts, right (they're also not yet available in Firefox but I'm sure that will change)? Will this be the case in insecure contexts too? I suppose browsers are moving away from allowing insecure sites at all.

[torokati44]

not yet available in Firefox but I'm sure that will change)?

Certainly, since it already works if enabled in about:config, it's just not yet stable/complete.

The VideoDecoder part of WebCodecs (and many other APIs) are only available in secure contexts, right

I have no idea. Anyway, we can keep the current solution if VideoDecoder is not available for whatever reason.

[torokati44]

I have decided that there is (currently) no need for any "internal" features such as h263, vp6, etc, within software_video, as this backend is currently only handling H.264 videos, on desktop. So, the single external_video added feature should suffice. Later, when a WebCodecs backend is added, that can be selected automatically, based on the target platform.

Also, the OpenH264 decoder currently drops the shared library, and the corresponding license, into the current working directory. I hope this satisfies the licensing terms about reproducing the text itself. The library is MD5 checksum checked if exist, but not repaired or replaced if it's not correct, it's simply not used. The license file is not repaired either, but it's also not even checked for contents if it exists.

I also thought about renaming the CLI argument to --enable-external-video instead, but keeping the required licensing notice. Any opinions on this? It would break the abstraction of which codecs are in which backends a little less.

[torokati44]

Also at the moment, the new, external video backend, when enabled as a feature, requires the existing software backend to be enabled too. There is no strict technical reason for this, it was just easier to do the fallback this way. For an MVP I think it's fine.

[torokati44]

Some more notes:

• There's still an assert or two left in there, I don't like that. No reason to panic when the decoder could also just stop working.
• Perhaps a proper H264Error enum would be nice, either only for decode-time errors, or maybe also for library-acquisition-time ones.
• When the NALU length is 3 or 4 bytes, the variant conversion could simple overwrite that with the start code. Even with 1 or 2 byte lengths, an in-vector shift would probably be better, as it can probably avoid a reallocation if there's spare capacity, plus it's nicer for cache locality.
• I ended up reusing the VideoDecoder trait from the software backend directly, instead of duplicating it just to add the configuration method. But then that call is still cut short right away in the software backend. Now it could be passed on to the decoders, where most of which will do nothing anyway. Just less abstraction breaking...
• What to so with the "stream end marker" packets? Do they matter? Maybe those too can be piped directly into the decoder?

[danielhjacobs]

At least for me on Fedora, this seems to have added a dependency, as I had to do this. If there's an equivalent dependency added to Ubuntu it should probably be noted in the README:

sudo dnf install libudev-devel

[Dinnerbone]

At least for me on Fedora, this seems to have added a dependency, as I had to do this. If there's an equivalent dependency added to Ubuntu it should probably be noted in the README:

sudo dnf install libudev-devel

libudev-dev is already listed, and (from my extremely ignorant Linux knowledge so I may totally be wrong) should be the equivalent package?

[sleepycatcoding]

At least for me on Fedora, this seems to have added a dependency, as I had to do this. If there's an equivalent dependency added to Ubuntu it should probably be noted in the README:

sudo dnf install libudev-devel

If I am remembering correctly libudev was introduced as a dependency by gamepad support (#14568).

[danielhjacobs]

Ah, I may have just not build the Desktop package from source since that PR was merged, my mistake.

[torokati44]

One last thing: I think I'll replace the test with a simpler, generated, "more test-y" video, that allows for accurate verification of frame order and timing.

[torokati44]

Now after #15974 this too needs to be updated. (And I guess an automated way to at least check weekly/monthly whether a newer OpenH264 release is out would be nice, but I'm not sure yet how exactly that would work.)

[torokati44]

Changed from bzip2-rs^1 to bzip2^2. Initially I used the former because I thought that the latter is only a wrapper for the library installed on the system. Actually, it thankfully bundles the bzip2 source code, and links it in statically, so no issues there!

~~Thank you for the async code, @J0sh0nat0r!~~

EDIT: Actually, I gave up on not blocking the main thread... :upside_down_face:

EDIT 2: Actually actually, it's kinda solved now, using tokio::task::block_in_place.

[torokati44]

I'm not entirely sure how enablement should be handled w.r.t. #16310... :thinking: ...as this is entirely desktop-specific.

[sleepycatcoding]

I'm not entirely sure how enablement should be handled w.r.t. #16310... 🤔 ...as this is entirely desktop-specific.

Maybe this should be in preferences instead and have an option to also toggle it in the UI?

[torokati44]

Maybe this should be in preferences instead and have a option to also toggle it in the UI?

Hahah, yeah, you're right, there just weren't preferences yet when I started working on it, and they didn't come to my mind just now... :sweat_smile:

[torokati44]

Made the toggle a preference now, instead of a command line option.

[torokati44]

There's now a bit of a back-and-forth between video/external: Add OpenH264 decoder and desktop: Add a preference to enable the OpenH264 decoder regarding license handling, but it doesn't bother even me that much to go through and rebase it out...

[torokati44]

Some further minor TODOs:

• ~~Remove or inactivate the toggle option in the preferences dialog if the feature is not enabled in the build.~~ DONE.
• The keyframe detection does not handle the cases when there are multiple actual NALUs clumped together in a chunk, and the IDR NALU is preceded by a SEI NALU (which is not that rare). Also the SPS/PPS NALUs could perhaps be considered "keyframes"...? Perhaps not?
• Change the "decoder did not produce a frame yet" from an Err to instead return a Result<Option<...>,...> from decode_frame, and make this case a None,

[Benman2785]

https://hoffmeister.li/tv/stream1.html

does NOT work :(

[lvyitian]

https://hoffmeister.li/tv/stream1.html

does NOT work :(

@Benman2785 This PR is DESKTOP-ONLY, it does NOT affect the web build of ruffle.

[lvyitian]

https://hoffmeister.li/tv/stream1.html

does NOT work :(

@Benman2785 The web implementation of H264 decoding will be based on WebCodecs API and is still work in progress.

[lvyitian]

https://hoffmeister.li/tv/stream1.html

does NOT work :(

@Benman2785 The reason is that if you don't use the OpenH264 prebuilt binaries provided by Cisco, you will have to pay for the patent fee to MPEG LA by yourself, and Cisco hasn't provided the WebAssembly-targeted prebuilt binaries of OpenH264, so we have to use the built-in decoders in the browser(which we haven't done yet).