wuffs
wuffs copied to clipboard
google
Audio and video codecs
Being able to decode audio and video would be nice. Audio and video decoders are a well-known source of security holes.
One problem is that many existing implementations resort to hand-written assembler code. I am not sure if Wuffs can hope to match their performance.
Audio would be nice. FLAC and MP3 is already listed on https://github.com/google/wuffs/blob/main/doc/roadmap.md
Video is a lot more work, but, yeah, it would be nice to have something like ffmpeg with much more memory safety guarantees.
As for hand-written assembler code, Wuffs' PNG decoder already uses CPU-arch specific intrinsics, which is roughly equivalent.
As for hand-written assembler code, Wuffs' PNG decoder already uses CPU-arch specific intrinsics, which is roughly equivalent.
Depends on the situation; there are cases where the compiler will do a bad job at stuff like instruction scheduling or register allocation, but I agree that asm should be a last resort. That said, making assembler code safe would be amazing, if perhaps out of scope.
Has there been any consideration for the API for these decoders.
Audio should be pretty simple, a config for communicating the sample rate, amount of channels, and bit depth. And perhaps an extension on the IO buffers to let them be used as ring buffers.
For video I don't think it's as simple as the image API augmented with audio. But I have no real experience with movie clip APIs to know what one should expect from them. Especially when you want to be able to use GPU acceleration for playback.
For video I don't think it's as simple as the image API augmented with audio. But I have no real experience with movie clip APIs to know what one should expect from them. Especially when you want to be able to use GPU acceleration for playback.
I consider GPU acceleration to be out of scope for this.
Has there been any consideration for the API for these decoders.
I haven't done any API design work yet. I was going to do it at the same time I started on an actual FLAC or MP3 decoder, but there's other work to do first.
In my research on video codex's I have found mention of this NUT mux format from ffmpeg. Maybe it is a small enough problem to be useful.
Other possible candidates for video is https://www.jonolick.com/home/mpeg-video-writer (256 lines of code!)
References
https://github.com/godotengine/godot-proposals/issues/4710 (has some nut details at the end)
