Yellow-Dog-Man
Replacing FreeImage with another image library for more file format support & faster image operations
Is your feature request related to a problem? Please describe.
FreeImage is a little slow compared to other image libraries for some operations. This isn't too terribly bad of a problem a lot of time, but I do generally enjoy when things are faster and snappier.
Describe the solution you'd like
libvips is an image processing library focused heavily on speed and low memory usage. It consistently performs better than FreeImage while also using less memory.
It would be cool to use libvips as a primary image backend for all the major formats, while FreeImage as a fallback for esoteric formats that vips may not support as well. For example, I know that libvips doesn't exactly have full support for RAW image formats like FreeImage does, so that might be one that needs handling by FreeImage.
Describe alternatives you've considered
N/A
Additional Context
Cyro has more experience specifically with the .NET bindings of the library, so he can give more insight from the perspective of using it in C#.
UPDATE: I prove a long and detailed writeup that serves as all the "additional context" here: https://github.com/Yellow-Dog-Man/Resonite-Issues/issues/4347#issuecomment-3508778314
Requesters
yosh, Cyro
We might look into the library for other reasons, but I'm a bit confused on the slowness.
What operations are you talking about exactly? Most of the time FreeImage doesn't really come into play - it's generally importing assets initially, but once they're imported it's not as involved.
Yeah. I was looking at decompiled code and realized that it wasn't used nearly as everywhere as I thought--mainly for importing/exporting files. I should've edited the issue text while writing it to reflect that. My bad.
Vips is fast at its operations on bitmaps, but importing/exporting files is delegated to, well, the outside libraries for those files. In that regard it's basically the same speed as FreeImage for Resonite's current use case (I benchmarked this with Cyro).
However, I do think it'd be cool to use though. For one, I see that the current operations on Bitmap2Ds are just done inside FrooxEngine by hand. It might be worthy to see if there's a performance improvement by delegating these operations to vips instead and wrap the byte array used in Bitmap2D with a vips image. Secondly, it's just... a more modern and less spiderweb-y codebase than FreeImage. I dunno.
I don't think we'd use it extensively for operations, since part of our philosophy is to have as much in the managed C# realm as possible - it also allows them to be defined more elegantly and easier to expand upon (and in the future provide bindings to implement your own too).
We might switch to it regardless for the import/export, but we'll see.
Something I didn't realize until raidriar just brought it up with me: If vips is used, we'd just get AVIF and JXL support for free--it'd just be able to do that. Which I think is neat.
Yeah. That's one of the reasons we were considering it. I believe it was suggested in the AVIF/JXL threads.
I don't think we'd use it extensively for operations, since part of our philosophy is to have as much in the managed C# realm as possible - it also allows them to be defined more elegantly and easier to expand upon (and in the future provide bindings to implement your own too).
We might switch to it regardless for the import/export, but we'll see.
We might want to use it to power some of the underlying operations actually. Part of the appeal is that vips is built so that it's image operations are pipelined, which makes them very fast. We could abstract our image operations so that whatever depends on them goes through our api -> vips -> result.
That way, we can take advantage of it's speed, but still easily shift away from it if we wanted to swap libraries again in the future by just changing the underlying implementation for a different, yet equivalent one.
I've been playing with the library for a while and I think there would be merit to using some of it's image manipulation primitives just because they're some of the fastest I've seen. It even beats out ImageSharp (considered also to be quite fast) by a considerable margin.
The mentioned RAW support that FreeImage has might not actually help soon. Newer devices are now encoding RAWs with JPEG XL internally, which FreeImage would naturally fail to decode anyway.
Me, Yoshi and Raidriar have been having extensive discussions on the Discord server for the past 3 weeks. Benchmarking performance and quality with different settings, exploring mods to test different libraries, ect.
Now that I'm aware this issue exists, I can give input too when needed. I actually know the lead libvips developer, who I worked with to add JPEG XL support to Bsky earlier this year.
Over the past 2 ish months, I have been working on experimenting with replacing FreeImage myself. Here's my findings:
Motivation
My initial motivation for even doing this was due to libvips being "fast" and "with a small memory footprint" and freeimage being "slow". However, a lot of the Bitmap2D operations in Resonite can be done very trivially in-place without needing to delegate them to an external library (channel swapping, removing channels, rotating, ...), and in fact FrooxEngine does them in C# land already. These trivial transformations are already fast enough as-is and wouldn't really benefit from doing them in native-library land, especially given how infrequently they are done.
The only transformation that is delegated to FreeImage is resizing images, which makes sense as it's not trivial.
However, the two bigger factors that I realized should be the motivation before working on my mod were:
- Wider image format support (namely: AVIF, HEIC, and JPEG XL)
- A generally more modern, flexible, and higher-quality codebase that gives a lot of freedom in the .NET side than what FreeImage currently does.
The first benefits users, while the second benefits YDMS.
The candidates
There's a whole host of image libraries in general, and I'm not going to list them all out here. However, the amount of candidates for what can replace FreeImage is a lot smaller because of file format support.
Now, FreeImage supports a lot of bespoke formats, many of which have no real need to be used in Resonite (to give an idea on what I'm talking about, FreeImage supports the kodak photo cd image format). If a replacement library needed to support every single format that FreeImage supported, there's only really two choices of image library replacements:
- ImageMagick, which is one of the oldest image libraries that basically supports every possible image format under the sun.
- libvips
- The only reason why libvips is a candidate here is because it has the ability to use ImageMagick to load any format that ImageMagick supports.
However, if the requirements for these bespoke image formats are relaxed a little, then OpenImageIO becomes another candidate to consider. When I say a "little", I really only mean those extremely obscure and bespoke formats. There are still a few formats that, while obscure, I have definitely seen used in Resonite. These include JPEG 2000, for example.
I didn't bother looking at ImageMagick because it'd be pretty obvious that it'd work as a replacement. I was less sure on libvips and OIIO, so I developed a mod as my staging ground for this stuff.
OpenImageIO
OIIO is an image library that is specifically intended for VFX/animation usages, a field that shares much overlap with game development/engines, including Resonite. It supports all relevant formats that FreeImage currently supports.
Advantages of OIIO include:
- Native support for 3d images
- Upstream manages some submodules that are included in the build system statically, which alleviates pressure on the team to manage dependencies on their end for some libraries. Not all libraries, though, especially if general image compatibility is desired.
Disadvantages of OIIO include:
- There's no C# bindings. I made a very thin set of bindings for OIIO myself that supports loading and writing out image data via the
ImageBufAPI, but these bindings are not what I'd call "battle-tested" against bespoke stuff like 3d images.
libvips
libvips is an image library with focus on speed and memory efficiency. As Cyro pointed out, it is "pipelined" and "demand driven" in the sense that image operations are never done until the image actually needs to be written somewhere. This allows the library to do certain optimizations that are otherwise impossible to do with conventional image libraries. If you look at their benchmarks for the native library as well as the c# bindings, it's wild stuff.
Advantages of libvips include:
- It is extremely fast and memory efficient. Using it in my mod has seen generally faster resizes when resizing images, even when ensuring that the compression factors are the same in the underlying external libraries.
- It already has very mature C# bindings.
Disadvantages of libvips include:
- There is currently no support for writing OpenEXR files natively through vips. You are able to write them when you compile it with an ImageMagick that supports OpenEXR, but I didn't test this in my compilation of ImageMagick as a dependency. This means that, when using my mod for testing, you are unable to resize EXR images in-game or export them using the export dialog.
- Funny thing though is that the JPEG XL file format supports HDR, so in theory it'd be possible to use JXL as an HDR file format when resizing EXR.
- Though, I might make an openexr saver for libvips myself and contribute it upstream. If that happens though, it won't be for a while as I want to focus on BCn compression stuff.
- The build system doesn't do any kind of included-library management for you when compiling on Linux. If you want a nice and easy shared library with all the dependencies statically compiled into it, you gotta do that yourself. More on this in the mod section.
- They do have a system for statically compiling windows DLLs easily, though. This would need to be modified a bit to statically contain all the dependencies that are needed for format compatibility.
My mod
This all brings itself to the mod that I made for replacing FreeImage. There are two branches in this repo:
mainuses purely libvipsoiio-vips-frankensteinis, well, a frankenstein where OIIO is used for importing/exporting, while vips is used for (most) underlying image operations- I stopped work on this branch because it was becoming quite evident that the operations done during export would be better off done in OIIO, which would negate the advantage I said of using libvips because it's fast. I don't recommend using this branch at all.
I also made a repo that houses scripts for building libvips on linux. This repo would (in theory) build vips in such a way that it requires 0 dynamic library dependencies (excluding stdlib stuff as well as expat, which is provided by the steam runtime container). However, for some machines, the build doesn't result in that. I'll need to eventually make a containerized version of it so the build can be 100% reproducible, but that's not my focus right now. This repo also doesn't leverage the full compatibility that ImageMagick provides, rather only using it for loading BMP files since vips doesn't natively support them.
The mod is currently in a state where I don't see any images failing to load, whether visually or when debug-printing them. I have also been able to import JPEG XL images in game using the mod. In theory, AVIF would also be able to be imported, but I was getting lazy and didn't bother including AVIF in my libvips build scripts, so my current version doesn't have AVIF support. I have been using this mod by default for the past few weeks just fine.
Conclusion
Overall, I do think libvips is the ideal replacement for FreeImage. OpenImageIO is also a very good candidate, but the lack of C# bindings really brings it down. As a complete last resort, ImageMagick can definitely be a replacement due to its wide support and existing C# bindings.
If it is decided that it's not worth supporting every single image format that FreeImage currently supports, content compatibility would need to be ensured. This would probably mean ensuring that every image stored in assets.resonite.com (via iterating over all record assets, extracting all StaticTexture2Ds, then seeing if they're resdb assets...) are in a format that would be supported by the new library, converting them losslessly if needed. Though I'm just spitballing that; it's fun to think about.