picom [feature request] more interface for shaders

problem description

the current interface for the glsl shaders doesn't provide enough features using window-shader-fg-rules, we can use more interface, but for n shader effects depending on n window properties in a boolean way, we need 2^n shaders to apply the rules. for example:

shader effect 1:

window borders, adds rounded borders to windows for window managers that don't have rounded borders themselves (like i3)
border color depends on focusedness of the window

shader effect 2:

fancy pants transparency, adds transparency depending on the color/brightness/etc of the window
don't apply to some windows

result: 4 shaders with a lot of redundancy:

focused_border+transparent
unfocused_border+transparent
focused_border+opaque
unfocused_border+opaque

proposed additions

uniform bool fullscreen;
uniform bool focused; probably also any other bool, int or float property could be useful to some people

by adding everything possible, the number of needed shaders could be reduced to a minimum

alternatives

adding functionality to pass any conditions as bool to shaders

would also be able to move any rules into the shader -> eliminates shader redundancy as much as possible
probably more complicated (if not impossible) to implement

or

be able to stack shaders (one shader per shader effect)

probably impossible

or

using window-shader-fg-rules disadvantages see problem description

May 16 '23 12:05 quantenzitrone

fancy pants transparency, adds transparency depending on the color/brightness/etc of the window

I wanted that feature so bad, so I wrote a little shader, so I'll put it here, so other people could find it:

#version 330

in vec2 texcoord;
uniform sampler2D tex;
const float min_alpha = 0.8;
const float upper_bound = 0.18;

vec4 default_post_processing(vec4 c);

vec4 window_shader() {
    vec2 texsize = textureSize(tex, 0);
    vec4 color = texture2D(tex, texcoord / texsize, 0);

    float pixel_brightness = (color.r + color.g + color.b) / 3;

    color.a = mix(min_alpha, 1, pixel_brightness / upper_bound);

    return default_post_processing(color);
}

the transparency depends on the "brightness" of the pixel (average of r,g,b)
it then interpolates between min_alpha and 1 to set the alpha of the pixel based on it's brightness
upper_bound is value of brightness that should be of alpha 1.0, anything higher too

screen screen

and a little caveat - I need to set a black background with feh (just a 1 black pixel image) before I set wallpaper with anything else (like wal, xwinwrap+mpv, nitrogen etc). xsetroot -solid "#000000" does not work, and feh --bg-fill leaks some buffer with bits of your screen.

feh --bg-max "$HOME/.bins/black_dot.png"

HOWEVER - this really doesn't play nicely with content like videos or photos. A lot of dark areas on them will leak your wallpaper

Jun 09 '23 20:06 barbiefan

i made a better version of your shader a while ago here: https://github.com/ikz87/picom-shaders there are also some other nice shaders in there

Jun 12 '23 08:06 quantenzitrone

i made a better version of your shader a while ago here: https://github.com/ikz87/picom-shaders there are also some other nice shaders in there

Great work! But it's overkill for what I was trying to achieve. Other shaders are also nice, However a lot of them (especially in 3d section) are segfaulting in picom

Jun 12 '23 13:06 barbiefan

I think #726 (loading custom textures) might be able to be reclassed as a subset of "more interface for shaders."

Jun 21 '23 23:06 wyatt8740

However a lot of them (especially in 3d section) are segfaulting in picom

do you have a nvidia gpu? some of them are indeed causing a segfault but for me the segfault happens somewhere in the nvidia's drivers so i don't think that we can do something about this.

sample backtraces

#0  0x00007f9693b6231c in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#1  0x00007f9693b66d2b in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#2  0x00007f9693b62318 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#3  0x00007f9693b6bd89 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#4  0x00007f9693b671d4 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#5  0x00007f9693b69fd6 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#6  0x00007f96952c5607 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#7  0x00007f96952c6bce in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#8  0x00007f9694b631a3 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#9  0x00007f9694b64215 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#10 0x00007f9694b7267f in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#11 0x00007f9694b48660 in ?? () from /usr/lib/libnvidia-glcore.so.535.54.03
#12 0x0000562fb8f69d59 in gl_create_program (shaders=0x562fbb0c8500, nshaders=4) at ../src/backend/gl/gl_common.c:82
#13 0x0000562fb8f6a0c3 in gl_create_program_from_strv (vert_shaders=0x7fffca719680, frag_shaders=0x7fffca719690)
    at ../src/backend/gl/gl_common.c:143
#14 0x0000562fb8f6b4fb in gl_win_shader_from_stringv (vshader_strv=0x7fffca719680, fshader_strv=0x7fffca719690, 
    ret=0x562fbb0a1250) at ../src/backend/gl/gl_common.c:593
#15 0x0000562fb8f6c36c in gl_create_window_shader (backend_data=0x562fbab8e020, 
    source=0x562fbaabcd10 "#version 430\n#define PI 3.14159265\n\n// These shaders work by using a pinhole camera and raycasting\n// The window 3d objects will always be (somewhat) centered at (0, 0, 0)\nstruct pinhole_camera\n{\n    "...)
    at ../src/backend/gl/gl_common.c:774
#16 0x0000562fb8f1a4c9 in initialize_backend (ps=0x562fbaab99b0) at ../src/picom.c:629
#17 0x0000562fb8f1ce9b in redirect_start (ps=0x562fbaab99b0) at ../src/picom.c:1374
#18 0x0000562fb8f1ba35 in paint_preprocess (ps=0x562fbaab99b0, fade_running=0x7fffca7198ac, animation=0x7fffca7198ad)
    at ../src/picom.c:1036
#19 0x0000562fb8f1e332 in draw_callback_impl (loop=0x7f9697874060, ps=0x562fbaab99b0, revents=256)
    at ../src/picom.c:1739
#20 0x0000562fb8f1e6eb in draw_callback (loop=0x7f9697874060, w=0x562fbaab9a70, revents=256) at ../src/picom.c:1802
#21 0x00007f96978690cb in ev_invoke_pending () from /usr/lib/libev.so.4
#22 0x00007f969786cd10 in ev_run () from /usr/lib/libev.so.4
#23 0x0000562fb8f23a3f in session_run (ps=0x562fbaab99b0) at ../src/picom.c:2740
#24 0x0000562fb8f23d9a in main (argc=4, argv=0x7fffca719ba8) at ../src/picom.c:2845

#0  0x00007f80e54f7990 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#1  0x00007f80e53332df in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#2  0x00007f80e5333043 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#3  0x00007f80e53332c8 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#4  0x00007f80e5338b49 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#5  0x00007f80e53371ed in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#6  0x00007f80e5338a9c in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#7  0x00007f80e533a103 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#8  0x00007f80e5313fad in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#9  0x00007f80e5318e2e in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#10 0x00007f80e5319825 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#11 0x00007f80e63ac975 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#12 0x00007f80e63af840 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#13 0x00007f80e63b40a6 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#14 0x00007f80e638c3d3 in ?? () from /usr/lib/libnvidia-eglcore.so.535.54.03
#15 0x000055d9dda3ed59 in gl_create_program (shaders=0x55d9dfc851a0, nshaders=4) at ../src/backend/gl/gl_common.c:82
#16 0x000055d9dda3f0c3 in gl_create_program_from_strv (vert_shaders=0x7ffd00b6f5c0, frag_shaders=0x7ffd00b6f5d0)
    at ../src/backend/gl/gl_common.c:143
#17 0x000055d9dda404fb in gl_win_shader_from_stringv (vshader_strv=0x7ffd00b6f5c0, fshader_strv=0x7ffd00b6f5d0, 
    ret=0x55d9dfc3a8f0) at ../src/backend/gl/gl_common.c:593
#18 0x000055d9dda4136c in gl_create_window_shader (backend_data=0x55d9dfab51b0, 
    source=0x55d9dfa09d10 "#version 430\n#define PI 3.14159265\n\n// These shaders work by using a pinhole camera and raycasting\n// The window 3d objects will always be (somewhat) centered at (0, 0, 0)\nstruct pinhole_camera\n{\n    "...)
    at ../src/backend/gl/gl_common.c:774
#19 0x000055d9dd9ef4c9 in initialize_backend (ps=0x55d9dfa069b0) at ../src/picom.c:629
#20 0x000055d9dd9f1e9b in redirect_start (ps=0x55d9dfa069b0) at ../src/picom.c:1374
#21 0x000055d9dd9f0a35 in paint_preprocess (ps=0x55d9dfa069b0, fade_running=0x7ffd00b6f7ec, animation=0x7ffd00b6f7ed)
    at ../src/picom.c:1036
#22 0x000055d9dd9f3332 in draw_callback_impl (loop=0x7f80e936e060, ps=0x55d9dfa069b0, revents=256)
    at ../src/picom.c:1739
#23 0x000055d9dd9f36eb in draw_callback (loop=0x7f80e936e060, w=0x55d9dfa06a70, revents=256) at ../src/picom.c:1802
#24 0x00007f80e93630cb in ev_invoke_pending () from /usr/lib/libev.so.4
#25 0x00007f80e9366d10 in ev_run () from /usr/lib/libev.so.4
#26 0x000055d9dd9f8a3f in session_run (ps=0x55d9dfa069b0) at ../src/picom.c:2740
#27 0x000055d9dd9f8d9a in main (argc=4, argv=0x7ffd00b6fae8) at ../src/picom.c:2845

Jul 10 '23 12:07 absolutelynothelix

do you have a nvidia gpu?

yeah, I do. I didn't realize that their drivers were this buggy, I thought bugs were more random and/or chaotic, but this seems to be stably reproducible.
Maybe we should file a bugreport on this one? I know that the scale of the problem isn't huge so they likely wouldn't care, but at least we would do our part?
Idk how to trace this on my machine tho

Jul 10 '23 20:07 barbiefan