rerun-io
Support images in BGR and BGRA
Describe the annoyance
There is generally no consensus between RGB and BGR image formats. For example, Pillow's PIL.Image.open() yields RGB format (including when the image is converted to a Numpy array). OpenCV's cv2.imread() or VideoCapture instead yield BGR numpy arrays.
rr.log_image() expects RGB and yields false colours otherwise. ~~At the very least, this should be documented.~~ Ideally, some optional format argument should be added so alternative formats may be correctly ingested.
To Reproduce This code yields false color (e.g. blue skins):
import cv2
import rerun as rr
rr.init("demo")
rr.spawn()
image = cv2.imread("image.jpg")
rr.log_image("image", image)
This is fixed by adding explicit format conversion:
# ...
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)
rr.log_image("image", image)
Ideally, something like that would be possible:
rr.log_image("image", image, format=rr.BGR)
FWIW np.array(mediapipe.Image.create_from_file("image.jpg").numpy_view()) is RGB
This is related to adding more semantics to color tensors in general:
- RGB, BGR, RGBA, …
- Linear vs gamma
- Data range (0-255, 0-65535, …) (https://github.com/rerun-io/rerun/issues/2341)
- Color map
The quick-fix is just to reorder the data into RGB on the SDK side.
The deeper fix is adding a "ColorSemantics" component or similar.
Note some places in the code indicate we should change our defaults to BGR.
I do NOT think we should do this. BGR is a bad, legacy choice from opencv. It should not be our default.
The simple way to implement BGR and BGRA right now is to upload them as RGB(A) textures, and add a swap_red_blue: bool flag to the rectangle.wgsl shader
