pygame integration: confusion over rgba byte order

[hanysz]

Looking at https://github.com/pygobject/pycairo/blob/master/docs/integration.rst -- this is a clear and helpful page. But following the instructions for pygame, I had issues with green and blue being invisible!

It turns out, at least on my system, that if you create a surface with surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, width, height) then the line image = pygame.image.frombuffer(buf, (width, height), "ARGB") will swap the channels, and you have to use image = pygame.image.frombuffer(data, (WIDTH, HEIGHT),"RGBA",) instead.

My best guess is that this is something to do with endianness, and pygame and cairo handling it differently. i don't know whether it would be consistent across different OSs and architectures, so can't say whether others would see the same results as me. Is it worth adding a comment to the documentation?

For reference: I'm on Ubuntu 18.04 64-bit, cairo.__version__ not found (how do I check my cairo version?), pygame version 1.9.1, python 2.7.1 and 3.6.9 (same behaviour on both).

[stuaxo]

It's certainly possible, though counter-intuitive, looking at the pygame docs https://www.pygame.org/docs/ref/image.html#pygame.image.frombuffer

Can you verify it by making a test image and loading it on the cairo side before displaying it ? It's worth making one where you label each channel its colour.

It looks like pygame has some support for pre-multiplied alpha, maybe we should extend the example here to let pygame know the alpha format too.

[hanysz]

You mean like this?

import pygame, time, cairo

WIDTH = 300
HEIGHT = 250
PYGAME_SWAP_CHANNELS = False
# If False, then green and blue are invisible in the pygame window
# If True, then all are visible, but red and blue swapped

def write_text(x, y, words, r, g, b):
   ctx.set_source_rgb(r, g, b)
     # change (r,g,b) to (b,g,r) for pygame to display correctly -- but then saved image is messed up
   ctx.set_font_size(20)
   ctx.move_to(x, y)
   ctx.show_text(words)

pygame.init()
window = pygame.display.set_mode( (WIDTH, HEIGHT) )
screen = pygame.display.get_surface()
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, WIDTH, HEIGHT)
ctx = cairo.Context(surface)
write_text(100, 30, "red", 1, 0, 0)
write_text(10, 200, "green", 0, 1, 0)
write_text(200, 200, "blue", 0, 0, 1)
surface.write_to_png("test.png")

data = surface.get_data()
if PYGAME_SWAP_CHANNELS:
  image = pygame.image.frombuffer(data, (WIDTH, HEIGHT),"RGBA",)
else:
  image = pygame.image.frombuffer(data, (WIDTH, HEIGHT),"ARGB",)
screen.blit(image, (0,0))
pygame.display.update()
time.sleep(5)

Yes, the saved test.png has the correct colours.

Oh, and I forgot to mention: even after solving the invisibility problem, I still have red and blue the wrong way round, so need to use (g,b,r) colours instead of (r,g,b). But that's easy enough to address.

[stuaxo]

Cheers, that demos this really well.

The implementation of image_frombuffer looks like it does byteswapping https://github.com/pygame/pygame/blob/main/src_c/image.c#L1092

Based on SDL_BYTEORDER, and SDL_CreateRGBSurfaceFrom which it calls may be able to do what we need, but there isn't an obvious way to do it.

I'm going to open a ticket on the pygame side as it's looks like it there's a missing feature here.

[hanysz]

Thanks, that's some nice detective work! As an end user, I've now got it doing what I need (I can "manually" swap the bytes), and am just looking to get this documented in case others trip over the same issue. But if you can actually get this fixed, that's even better. And it's interesting to see what's happening under the hood.

[stuaxo]

When I'm lucky enough to have the time, it's good to grab enough info so the developer on the other end doesn't need to think to much.

We'll keep this ticket open until we see what happens on the pygame side.

[stuaxo]

This has highlighted a couple of things:

• Every example should go for in both directions.
• Every example should be tested to ensure the output colour matches input.

[ldo]

I think Cairo always expects B to be in the least significant byte, G in the next byte up from that, and R in the byte up from that. In other words, given (r, g, b) components where each is a real in [0, 1], the Cairo pixel can be computed as

def to_pixel(c) :
    return \
        (
            round(c.r * 255) << 16
        |
            round(c.g * 255) << 8
        |
            round(c.b * 255)
        )
#end to_pixel

and this calculation is endian-independent.

[rlatowicz]

At the moment, a fix is to use PIL (or something) to do the conversion BGRA (cairo surface) to RGBA (pygame image),

The above code with an example fix,

import pygame, time, cairo
from PIL import Image

WIDTH = 300
HEIGHT = 250
PYGAME_SWAP_CHANNELS = False
# If False, then green and blue are invisible in the pygame window
# If True, then all are visible, but red and blue swapped

def write_text(x, y, words, r, g, b):
   ctx.set_source_rgb(r, g, b)
     # change (r,g,b) to (b,g,r) for pygame to display correctly -- but then saved image is messed up
   ctx.set_font_size(20)
   ctx.move_to(x, y)
   ctx.show_text(words)

pygame.init()
window = pygame.display.set_mode( (WIDTH, HEIGHT) )
screen = pygame.display.get_surface()
surface = cairo.ImageSurface(cairo.FORMAT_ARGB32, WIDTH, HEIGHT)
ctx = cairo.Context(surface)
write_text(100, 30, "red", 1, 0, 0)
write_text(10, 200, "green", 0, 1, 0)
write_text(200, 200, "blue", 0, 0, 1)
surface.write_to_png("test.png")

# using PIL, convert from BGRA to RGBA
data = Image.frombuffer('RGBA', 
                        (WIDTH, HEIGHT),
                        surface.get_data().tobytes(),
                        'raw', 'BGRA', 0, 1).tobytes()

image = pygame.image.frombuffer(data, (WIDTH, HEIGHT),"RGBA",)

screen.blit(image, (0,0))
pygame.display.update()
time.sleep(5)

A direct pycairo <-> pygame format would be ideal.

[rlatowicz]

It should also be noted that at present, the demo, pygame-demo.py

and the pycairo docs, Pygame & ImageSurface

are incorrect.