[Feature request]: Cross platform screen capture using xcap

[bo0tzz]

Please describe your feature request

xcap (https://github.com/nashaofu/xcap) seems to have a nice and simple API for cross-platform screen capturing. I was hoping to implement this myself in order to get support for my X11 environment, but I'm way out of my depth and can't work out how to feed the captured frame into Vulkan::luma_percent. If anybody has pointers on that front I'd still be happy to try my hand at a PR.

[max-baz]

Hello!

Before talking about implementation details, I want to highlight a big caveat: it looks like xcap will get the screenshot out, which you can then process. When I tried that approach, it was extremely power-hungry and had a significant impact on my laptop's battery life (and heat generation). In order to reduce the load, it will be tempting to reduce the frequency of taking & processing screenshots, but it would result in wluma being very slow to react to you changing windows, for me personally that was very annoying and a deal-breaker, and that was the reason for me to look into implementing it on GPU.

If you have a wlroots-based environment, where you can run wluma with wlroots capturer, let me know, because it is easy to adapt the code to simulate the "screenshot-based" approach - you might want to try to spend a day in that "simulated" environment, to see if you find it acceptable or not, before spending more time trying to integrate xcap.

Having said all that, the answer to your question is that you don't need to plug that into Vulkan at all (Vulkan is there to keep & process that frame in GPU, without extracting it in a form of screenshot). With xcap, you would just add a new file src/frame/capturer/xcap.rs, where your job would be to periodically take a screenshot, compute its perceived lightness (e.g. just passing the entire image to this function), and call controller.adjust(), passing the computed "lightness" - the skeleton is in none.rs file, which doesn't process any screenshots and simply passes 0 all the time:

https://github.com/maximbaz/wluma/blob/08b48960cbb8325a8546d51aadc3ed5461747e74/src/frame/capturer/none.rs#L8-L13

I do encourage you to try, just want to manage your expectations :grin:

[max-baz]

See also https://github.com/maximbaz/wluma/issues/5 for a quite promising alternative

[bo0tzz]

When I tried that approach, it was extremely power-hungry and had a significant impact on my laptop's battery life (and heat generation).

That's a great point that I failed to consider because I'm on a PC :sweat_smile:

Thanks for the reference to compute_perceived_lightness_percent, I somehow managed to miss that. I think I got it to work with that, although I'm very much hacking my way around:

impl super::Capturer for Capturer {
    fn run(&self, _output_name: &str, controller: Controller) {
        let monitors = xcap::Monitor::all().unwrap();
        let monitor = monitors.first().unwrap();
        let controller = Rc::new(RefCell::new(controller));

        loop {
            let frame = monitor.capture_image().unwrap();
            let (x, y) = frame.dimensions();
            let pixels = x * y;
            // Absolutely no idea whether this is the right call
            let rgbas: &[u8] = frame.as_bytes();
            let luma = compute_perceived_lightness_percent(rgbas, true, pixels as usize);
            log::trace!("Luma: {}", luma);
            controller.borrow_mut().adjust(luma);
            thread::sleep(std::time::Duration::from_millis(200));
        }
    }
}

This compiles and runs (good start lmao), and the luma value gets higher for brighter screen content, though I have no idea how to tell whether the actual value makes sense (I get about 3 for a dark terminal, and 80 for a light mode website).

I did read #5, and it definitely sounds like the right way to approach this, but it feels way out of my reach to try to implement so I took the easy road.

[bo0tzz]

I can see what you meant, lol

[max-baz]

A few weeks later, I'm curious if you still use wluma with your patch, even though it consumes an entire CPU core? I suppose it's less noticeable on desktop than on laptop, but I imagine you might be feeling the effects when you do some other heavy operations?

[bo0tzz]

No, I conceded that it's a bad idea :sweat_smile: I'll close this issue as well, since clearly it doesn't make sense to really implement.