Y coordinate of texture from [pix_snap] is flipped in shaders when texture is given via texunit

[php4fan]

Detail

This bug didn't exist around 2012. However, I tried to compile old snapshots of Gem from github and went as far as December 13 2018, and the bug is already present there. I was unable to compile older versions due (I think) to incompatibilities with the current version of Imagemagick.

Steps to reproduce:

1. unzip the attached file: pix_snap_glsl.zip
2. open REPRO.pd

This is basically a combination of example 10.glsl/01.simple_texture.pd and the help patch for [pix_snap]. It works like the latter, but the snapshot image is processed with the shader from the former.

3. turn on rendering
4. click one of the two bangs "load shader" or "load another shader".

EXPECTED BEHAVIOR

The blue sphere "painted" on the white square should rotate in the same direction (counterclockwise) as the red sphere.

OBSERVED BEHAVIOR

The blue sphere goes clockwise, opposite to the red one. This is because the image generated by pix_snap, as seen from the shader, has the y coordinate flipped.

COUNTEREXAMPLE

Now open NO_REPRO.pd and try the same.

This one does not reproduce: in this case the two spheres rotate in the same direction. The y coordinate of the texture seen by the shader in this case is not flipped.

The difference between the two patches is that NO_REPRO.pd "feeds" the pix from pix_snap into the shader "directly" by simpli connecting them along the same "chain" from [gemhead].

In REPRO.pd, on the other hand, [pix_snap] and [glsl_program] are on different chains, the texture that gets the image from [pix_snap] is assigned a [texunit( and the same value is given to the shader via the [MyTex( uniform variable referencing the same texunit.

For some reason, only the latter situation triggers the bug.

I'm pretty sure I'm using all the wrong terminology, sorry about that.

PC Vs Mac

I am only testing on Linux now. However, when I wrote the original patches that are now affected by this issue, I was working on both Linux and Mac, and IIRC I found out that shaders on Mac would always have the y coordinate of all textures (not only those coming from pix_snap) consistently flipped upside-down. That is, I had to write two versions of my shaders, one for Pc (I seem to remember it worked the same on Linux and Windows) and the other for Mac, where the Mac version had to flip the y coordinate (of course I could have written a unified shader and used some uniform variable to tell it whether or not to flip the y). I don't remember whether that was documented and I don't know whether that is still true. Either way, that seems like something that might interfere with testing and/or fixing this issue on Mac, so take that into account.

Gem Version

0.94.git v0.94-1079-g92089a6ff

Pd Version

0.55-2

Operating System

Linux

OS Version

Manjaro

Which CPU are you using?

amd64/x86_64 ("64bit Intel")

[php4fan]

I have been able to narrow this down to a very specific issue, so I have completely rewritten the report, and attached a much more minimal reproducing patch.

If you had already read the original report, please re-read. If you hadn't, even better.

[php4fan]

This looks kind of suspicious?

https://github.com/umlaeute/Gem/commit/3664cf4610d87fdcc0fbfcfaa5679157ad4e0be1#diff-408f63bac050ac408d9377ea39c4cd11f70c8540dfa737aa824e379001d4e16fR108

[umlaeute]

openGL's image orientation uses the bottom/left corner as the 0/0, whereas practically all pixel acquisition methods (image readers, video capture devices,...) use the top/left corner as the origin.

traditionally, Gem tried to not correct the upside-down orientation when pixel data is read - for performance reasons. instead the data is flipped using texture coordinates (within [pix_texture]).

what did happen in the last years, is that we now use top/left uniformly across the various "external" pixel sources (so we stil do not flip the image, but at least virtually all pixes are "upside down", and we can handle them all the same).

afaik the only notable exception to this is... [pix_snap]. the reason is, that [pix_snap] reads the openGL buffer, which is (according to openGL) bottom/left oriented.

and again: for performance reasons we do not flip the image data in main memory: applying the transformation in GLSL is super-simple (and much faster).

so rather than using two different shaders, I think the proper way is to just use a uniform variable to tell the shader whether it should flip the image or not. you can get the required information either from [pix_texture]s right outlet (as well as from [pix_snap2tex]). or use [pix_info].

if you prefer to flip the data via the CPU, you could also add a [pix_flip] object after [pix_snap]. [pix_flip] will normalize the upside down orientation (so pixes coming out of if will always use top/left orientation). so if you tell it "not to flip" (the default), it will be a no-op for pixes that are top/left oriented, and it will perform a vertical pixel data flip for pixes that are bottom/left oriented (unsetting the upsidedown flag). (if you tell it to do a "vertical flip", it will only fix the upsidedown flag for bottom/left pixes; whereas top/left pixes get a pixel data flip).

so i think the answer is: "yes, i know. but this is a won't fix".

[php4fan]

Oh, so you are just happy with breaking bc here, ok.

How do you explain the discrepancy between the two attached patches REPRO.pd and NO_REPRO.pd, i.e the fact that the y coordinate only gets flipped when assigning a texunit number and passing it to glsl_program and not when using glsl_program on the "default" texture coming from the gem chain? How should I deal with that?

[umlaeute]

Oh, so you are just happy with breaking bc here, ok.

nah. it was always broken.

however, most of the pixel sources have been fixed, which imho gives a net-improvement. (iirc, you had problems with needing different shaders on macOS/Windows/Linux; this was because different pix sources would use different orientation; now they all use the same, with a single exception)

[umlaeute]

How should I deal with that?

the simple answer is: insert a [pix_flip] somewhere between [pix_snap] and [pix_texture].

(iiuc, you (indirectly) propose that [pix_snap] should output the image in a ready-to-use format; this boils down to using a [pix_flip] equivalent internally. so to fix the problem, just us [pix_flip] explicitly)

[php4fan]

Ok but I'm still confused about this:

How do you explain the discrepancy between the two attached patches REPRO.pd and NO_REPRO.pd, i.e the fact that the y coordinate only gets flipped when assigning a texunit number and passing it to glsl_program and not when using glsl_program on the "default" texture coming from the gem chain?

although it's more of a curiosity at this point, because for now I don't need to mix the two scenarios.

[umlaeute]

although it's more of a curiosity at this point, because for now I don't need to mix the two scenarios.

[pix_texture] will store the texture orientation in the current gemstate and [square] will use that info to properly render it. however, the gemstate does not propagate from one [gemhead] to another, so the information is lost.