Encoder/decoder performance

[pps83]

Looks like encoder/decoder performance are the main issues of flif. As it was inspired by ffv1, is it common enough with ffv1 to be implemented using libavcodec to get all their perf optimizations? This might solve these performance issues with flif.

[pps83]

In general I got pretty good results (comparing flif encoded images to other codecs), however, by looking at the code I don't think it has a good chance for wide adoption. Try to implement it as a codec (without using libflif) for libavcodec and you'll increase your chances greatly. I tried to profile code and have some quick comments.

1. Code looks like mess. Take a look at a function that takes inclusive 75% of time flif_encode_FLIF2_inner. It's pretty bad and difficult to read, it's totally hostile to any debugging attempts with all these if one-liners. On top of that, design is quite questionable. For example, flif_encode_FLIF2_inner is templated over IO and other stuff... I find it very very questionable.
2. Looking at profiler results I see something that tells me that there is room for improvement (e.g. something isn't done right): Seems like heap is being abused heavily, taking overall over 9% just for 2 heap related functions that ended up in top 10. Division is taking almost 10% overall! Vector constructor takes 2.2% overall, but inclusive sample count is whopping 11.85%! Does your encoder really needs that kind of dependency on vector constructor?
3. Because of template abuse perhaps code size for decoder is about 1.1MB, webp decoder for comparison is around 300KB. I think if flif was properly written it should be less than webp decoder. Also, DECODER_ONLY build of flif saves just 200KB or so, as if all complexity of the codec is in the decoder, which is nonsense.
4. I didn't try to build for arm, but if doesn't do well on mobiles, then it doesn't have chances for wide adoption IMO: it might be stuck in endless loop: interested people will be waiting when decoder speed on mobiles becomes acceptable and decoder speed on mobiles won't be even looked at as if it's something low priority.

It's easy to add flif support to libav/ffmpeg by simply using libflif, but if you actually try to implement it directly in libavcodec then there is no way such code will ever be acceptable in ffmpeg. AFAIK, vp8 decoder was originally implemented in ffmpeg without using libvpx and it was much faster that libvpx and perhaps it's still is. Maybe you can at least implement decode in ffmpeg, otherwise it's hard co convince anybody to adopt it if decoder has performance issues.

Out of curiosity. In the updateChances function. bits=10, does that try to do 10bit per pixel lossless encoding for a png that was originally 8bit pp?

[lehitoskin]

I'm not sure I understand why you're talking about libav, a video format library, when flif is an image format. It would make more sense to me if you were talking about ImageMagick or some similar project, but why libav/ffmpeg? Also, it doesn't seem right to me for a reference image format library to depend on a video library for either decoding or encoding.

[pps83]

Any image is a video with a single frame. avcodec (libav/ffmpeg etc) handles perfectly all common image formats and many obscure ones. And on top of that it's extremely well optimized for common codecs (for x86/x64 and arm CPUs). I was surprised that ffmpeg wasn't able to open flif files, that's why I mentioned it. For example, to convert jpeg or png to webp: ffmpeg -i file.jpg file.webp

I'm not saying that flif should depend on ffmpeg, but if ffmpeg had flif decoder support implemented directly, then many applications would be able to use it as-is. Long time ago chrome/firefox used libavcodec for media stuff. ffmpeg also is faster to encode/decode jpeg and most of other formats than reference libraries. It's usually way better for other cpus (like arm).

I just did a quick look, and yes, chrome and firefox both use ffmpeg for media and most likely even for showing jpeg and png they use avcodec to decode them. They definitely use custom ffmpeg config (only with some codecs enabled), and if ffmpeg had flif support for them enabling flif support would be a matter of adding --enable-decoder-flif for avcodec configure plus some changes in code base.

[jonsneyers]

Looks like encoder/decoder performance are the main issues of flif. As it was inspired by ffv1, is it common enough with ffv1 to be implemented using libavcodec to get all their perf optimizations? This might solve these performance issues with flif.

FLIF was inspired by FFV1, but it is quite a bit more complicated and different. So no, I don't think it's common enough. But yes, there is a lot of room for perf and code design improvement, I agree with that.

Out of curiosity. In the updateChances function. bits=10, does that try to do 10bit per pixel lossless encoding for a png that was originally 8bit pp?

If the original is 8-bit RGB, then the default color transform goes to 9-bit YCoCg, and the actual numbers that are encoded are predictor residues which can be 10-bit. So for that reason the symbol encoder has to be able to handle 10-bit numbers if the original is 8-bit, and 18-bit numbers if the original is 16-bit.

[jnorthrup]

@pps83 writes "Try to implement it as a codec (without using libflif) for libavcodec and you'll increase your chances greatly."

there's not much more to say than this. flif video transcodes are definitely worth exploring, if the effort is the same as ffv1 to play with it.

[jnorthrup]

as it happens, i use ffmpeg in production to convert pngs to jpeg, webp, etc. to satisfy the lighthouse checkboxes for at least one client.