RGB images get converted to RGBA images before getting uploaded to the GPU,which is causing a huge performance drop

[zrezke]

Description

Displaying RGB images for the first time is very expensive as they get converted to RGBA before being sent off to the GPU. I did some profiling with a 4k video stream where this issue really becomes apperant:

Suggestion to resolve this issue

Instead of using the TextureFormat::Rgba8Unorm, maybe we could use a texture format like TextureFormat::R8Unorm and upload the RGB data directly to the GPU inside of this texture, then do some extra work in the shader to correctly sample an R8Unorm as an rgb texture.

If this performance issue get's resolved this would be really useful especially for live streaming data, where you really want to minimize the time wasted loading and converting the data...

[emilk]

This looks surprisingly slow to me - is this a debug build?

[zrezke]

Not sure what I was running at the time, I tried it again today with a fresh pip install:

rerun --version
rerun_py 0.6.0 [rustc 1.69.0 (84c898d65 2023-04-16), LLVM 15.0.7] x86_64-unknown-linux-gnu release-0.6 643dea9, built 2023-05-25T20:35:24Z

The times for pad_rgb_to_rgba range from ~6.5 ms to ~13 ms.

[Wumpf]

we moved all the image conversion to re_renderer now since some of those are gpu driven (see #7700) we should try to make this gpu driven as well: This is absolutely trivial if we support random buffer access (which is related to compute shader support and as such is not available on webgl): just upload the buffers and write everything out in a fragment shader (yes could be a compute shader, but why bother! fragment shader is likely faster for this operation). Doing this with textures all the way can be done, but not only does this likely have more overhead, we also may easily hit the max texture size limit way earlier since we'll need to update this as an R8/R16/R32(u/i/f) texture that has 3x the width