Devolutions
Add QOI compression
I recently found the QOI image format (https://qoiformat.org/) and was excited by its potential for desktop remoting.
I hacked up a few things to actually benchmark it compared to the other codecs.
Based on a raw recording of a typical Fedora desktop usage (terminal/web), for 730 4k frames:
None: 5s user CPU, 0% compression Bitmap: 74s user CPU, 92.5% compression RemoteFx (lossy): 201s user CPU, 96.72% compression QOI: 10s user CPU, 96.20% compression UPDATE: jpeg-turbo 80% 4:2:2: 16s user CPU, 97.92% compression jpeg-turbo 95% 4:4:4: 22s user CPU, 96.13% compression jpeg-turbo lossless: 135s user CPU, 86.06% compression UPDATE2: tile diff+QOI+zstd stream long window: 11s user CPU, 99.76% compression!!
There is still a lot of various optimizations to be desired regardless of the codecs, but this will hopefully help. Also, we should start thinking how QOI could be adapted for streaming.
Custom codecs not supported by the Microsoft RDP implementation have rarely been used, but since we're building both the client and server in IronRDP, we're free to experiment! QOI is an excellent choice, it is very simple, and easier to optimize due to its low complexity. IIRC QOI uses RGB, not YUV like most codecs. Color conversion and chroma subsampling alone can take up a lot of the processing time if not properly optimized.
There is just one custom codec extension I've seen in the past: JPEG in RDP. XRDP supports it in the server, and FreeRDP supports it in the client as an optional build feature. JPEG has many advantages: it is by far the most widely available image codec, but it is also one for which the most effort has been put into optimizing the implementation (think libjpeg-turbo). For an RDP web client, this also makes it possible to leverage built-in browser support for JPEG decoding, which is a huge advantage over porting and optimizing a custom codec in WASM.
In the past, we've designed our own remote desktop protocol (Wayk Now) at Devolutions, and we experimented with many different ways to deal with the codecs more efficiently than RDP. We had GFWX ported in Rust: https://github.com/Devolutions/gfwx-rs
GFWX, unlike QOI, is a simplified wavelet codec, which would make it closer to RemoteFX. It is generic and can accept a wide variety of YUV color formats (RGB wouldn't perform well, you're better off doing the YUV conversion to avoid the color channel correlation issue of RGB). It is one of the rare codecs that would enable really nice things like lossless encoding in which you could trim part of the encoded output for lossy encoding. In our implementation, we've used a reversible color conversion (YCoCg-R) due to its reduced complexity (it's only integer operations, not floating point). Unfortunately, we found out JPEG still performed better than GFWX. In theory GFWX offers ways to beat JPEG, but libjpeg-turbo is just hard to beat.
If you're curious about the color transformations we've experimented with, we still have them here: https://github.com/Devolutions/cadeau/tree/master/libxpp
I'm open to experimenting with various custom codecs, the ones in RDP are good, but not necessarily special or great. We can definitely beat the standard protocol if we go custom :)
@awakecoding I added some jpegturbo benchmarks above. For lossless, desktop usage, QOI is very good - jpeg is far behind. Performance with wasm should be ok (rust-qoi compiles for wasm target)
We used to have some heuristics in Spice server to detect video zones, and used jpeg aggressively iirc (when not using whole frame gpu video encoding).
I’m impressed by the characteristics of this codec, especially given how simple the compression format is.
As a custom codec, it seems to be a solid alternative to JPEG:
- Smaller CPU-usage than RemoteFX
- No dependency on jpeg-turbo, i.e.: smaller binary and less unsafe code
- Compression level on par with both RemoteFX and JPEG
And the Rust implementation is not even optimized yet.
For the reasons explained by Marc-André, I don’t think we can beat JPEG in the browser (yet?) even with WASM, but I suspect it’s better than the current implementation which does not rely on the native JPEG implementation of the browser anyway (RLE / RDP 6.0 compression codec → RGB → JavaScript Image object). Definitely does not hurt to use QOI. If we really want to squeeze performance using JPEG, we could consider that too in the future though.
Obviously the limitation is that only IronRDP ↔ IronRDP will be supported, but I’m interested in seeing this as an option similar to how JPEG is an option for FreeRDP ↔ XRDP. I’ll be glad to merge the PR when it’s ready.
By the way, do you have an idea how good it performs on the client side (decoding)?
Good opportunity to get some some basic benchmarks merged too.
Small suggestion: what about a benches/ folder at the root of the repository?
@CBenoit I think we should be good to start reviewing this series. I can split various preliminary patches in different MR if you prefer?
@CBenoit I think we should be good to start reviewing this series. I can split various preliminary patches in different MR if you prefer?
Sounds great! Yes, I would prefer multiple MRs. I’ll do my best to be as responsive as possible!
@CBenoit yes, this has been pending for too long. I was waiting for qoi-rust maintainer. To unblock the situation, I decided to fork qoi and released it on crates.io. The qoi-rust maintainer now said he will resume his work, but we can already make progress on this PR without waiting for him. thanks
Sounds great! I’m starting the review.
First, this commit is not following the convention: benches:: fix could not find time in tokio
I guess this should be chore(bench): or chore(benches) (tool change, do not go into production)