State of h264 VideoEncoder support

[Angramme]

Hey, again, I'm really giving you a hard time with the issues. I am trying to get a polyfill for VideoEncoder for the avc codec. I though that I'm not building or including libav correctly but that has been solved in the other issue. I consistently get "codec not supported" errors. I have looked through the built files for the webcodec-polyfill and found this:

function encoder(codec, config) {
        if (typeof codec === "string") {
            const codecParts = codec.split(".");
            codec = codecParts[0];
            let outCodec = codec;
            const ctx = {};
            const options = {};
            let video = false;
            switch (codec) {
                // Audio
                case "flac":
                    ctx.sample_fmt = 2 /* S32 */;
                    ctx.bit_rate = 0;
                    if (typeof config.flac === "object" &&
                        config.flac !== null) {
                        const flac = config.flac;
                        // FIXME: Check block size
                        if (typeof flac.blockSize === "number")
                            ctx.frame_size = flac.blockSize;
                        if (typeof flac.compressLevel === "number") {
                            // Not supported
                            return null;
                        }
                    }
                    break;
                case "opus":
                    outCodec = "libopus";
                    ctx.sample_fmt = 3 /* FLT */;
                    ctx.sample_rate = 48000;
                    if (typeof config.opus === "object" &&
                        config.opus !== null) {
                        const opus = config.opus;
                        // FIXME: Check frame duration
                        if (typeof opus.frameDuration === "number")
                            options.frame_duration = "" + (opus.frameDuration / 1000);
                        if (typeof opus.complexity !== "undefined") {
                            // We don't support the complexity option
                            return null;
                        }
                        if (typeof opus.packetlossperc === "number") {
                            if (opus.packetlossperc < 0 || opus.packetlossperc > 100)
                                return null;
                            options.packet_loss = "" + opus.packetlossperc;
                        }
                        if (typeof opus.useinbandfec === "boolean")
                            options.fec = opus.useinbandfec ? "1" : "0";
                        if (typeof opus.usedtx === "boolean") {
                            // We don't support the usedtx option
                            return null;
                        }
                        if (typeof opus.format === "string") {
                            // ogg bitstream is not supported
                            if (opus.format !== "opus")
                                return null;
                        }
                    }
                    break;
                case "vorbis":
                    outCodec = "libvorbis";
                    ctx.sample_fmt = 8 /* FLTP */;
                    break;
                // Video
                case "av01":
                    video = true;
                    outCodec = "libaom-av1";
                    if (config.latencyMode === "realtime") {
                        options.usage = "realtime";
                        options["cpu-used"] = "8";
                    }
                    // Check for advanced options
                    if (!av1Advanced(codecParts, ctx))
                        return null;
                    break;
                case "vp09":
                    video = true;
                    outCodec = "libvpx-vp9";
                    if (config.latencyMode === "realtime") {
                        options.quality = "realtime";
                        options["cpu-used"] = "8";
                    }
                    // Check for advanced options
                    if (!vp9Advanced(codecParts, ctx))
                        return null;
                    break;
                case "vp8":
                    video = true;
                    outCodec = "libvpx";
                    if (config.latencyMode === "realtime") {
                        options.quality = "realtime";
                        options["cpu-used"] = "8";
                    }
                    break;
                // Unsupported
                case "mp3":
                case "mp4a":
                case "ulaw":
                case "alaw":
                case "avc1": // < ---------------------------------------------    ME SAD :( 
                    return null;
                // Unrecognized
                default:
                    throw new TypeError("Unrecognized codec");
            }

It seems like the avc1/h264 codec is supported by libav but not the polyfill. Is that right? Is this currently being worked on or should I not expect support any time soon?

(PS: I really wanted something widely supported because my app targets unexperienced audience that would have trouble opening other codecs. [for example vp9 and av1 are not supported out of the box on safari, quicktime, and some social media apps] Unfortunately it seems that all good i.e. non-MPEG options are fairly new.)

[Yahweasel]

The bigger issue you're going to hit is that it's too damned slow to do video encoding in WebAssembly X-D

To quote the README: «FFmpeg supports many codecs, and it's generally easy to add new codecs to libav.js and LibAVJS-WebCodecs-Polyfill. However, there are no plans to add any codecs by the Misanthropic Patent Extortion Gang (MPEG), so all useful codecs in the WebCodecs codec registry are supported.»

If you want support for codec configurations for H.264, you will need to add them yourself. Pull requests are accepted.

[Angramme]

it's too damned slow to do video encoding in WebAssembly

Does that apply to all codecs or just MPEG codecs? Also this is still my fastest alternative. MediaRecorder will produce unusable choppy video in my case since it was made for real time recordings. Whammy, CCapture and alike all use frame by frame canvas exports which is just unusable speed wise.

Which codec (already supported by this polyfill) would you recommend to optimise for export speeds?

[Yahweasel]

it's too damned slow to do video encoding in WebAssembly

Does that apply to all codecs or just MPEG codecs?

All codecs listed by the WebCodecs codec registry. Video decoding is fast enough in enough cases, and I use polyfill VP8 for those cases, but to make video encoding of modern codecs acceptably fast, you'd probably need to combine threads (which libav.js is capable of but have severe restrictions on the surrounding page) and SIMD, and that SIMD would have to be custom-written for WebAssembly. There simply isn't the motivation in the community to do that.

Also this is still my fastest alternative. MediaRecorder will produce unusable choppy video in my case since it was made for real time recordings. Whammy, CCapture and alike all use frame by frame canvas exports which is just unusable speed wise.

For export, other than (obviously) using normal WebCodecs on Chrome and Safari, this is indeed your only option on Firefox. I don't know what your exact use case is, but if you want to tell users "exporting on Chrome and Safari are fast, and if you're exporting on Firefox, Idonno, come back tomorrow?" then encode however you'd like. Seems like a lot of work given that Firefox is in the process of implementing WebCodecs for video literally as we speak.

Which codec (already supported by this polyfill) would you recommend to optimise for export speeds?

If you want fastish video encoding in the polyfill today, you will have to step outside of codecs in the registry. Probably use H.263 in the form of DivX (libav.js encoder "mpeg4") with threads. This is why the polyfill supports bypassing the codec registry and using unlisted codecs.

But really, you need to calibrate your expectations. The state of play today is: audio en/decoding is fast using whatever, and video decoding if done carefully can be fast enough for many use cases; regularizing these is why the polyfill exists. But, if you need to encode video, you need to use browser WebCodecs. Meanwhile, Firefox and Safari only appear to be implementing the video side of WebCodecs, and Firefox hasn't finished encoding yet (or revealed any of this without a feature flag). MediaRecorder can, as you point out, be used for realtime encoding, but isn't reliable for rendering.

Assuming based on your discussion that what you're doing involves media rendering and export, my advice is this: use browser WebCodecs for video encoding and tell Firefox users they're just going to have to wait (and bear in mind that Firefox may very well never implement H.264 in WebCodecs for ethical reasons). Use the polyfill to regularize audio and possibly video decoding across platforms. Assuming you're creating actual media files, use libavjs-webcodecs-bridge to mux. Be prepared to adjust your (video) output options to whatever codecs are actually supported by the browser, and tell users "if you want H.264, you will need to use a different browser".