Support ScreenCapture loopback

[Kree0]

#876 I added a new host: ScreenCaptureKit in macos, which only provides input devices for capturing the sound of the Display. When I test it in KVM virtualized mocOS-14, it captures sound intermittently, I don't know why this happens, can anyone help me check the code?

[louis030195]

any update on this? need asap for https://github.com/louis030195/screen-pipe

how can i help?

i just tried this PR to record screen to wav and it works

https://gist.github.com/louis030195/013fda2fa6b373f815bbeeb39e524090

[Kree0]

any update on this? need asap for https://github.com/louis030195/screen-pipe

how can i help?

i just tried this PR to record screen to wav and it works

https://gist.github.com/louis030195/013fda2fa6b373f815bbeeb39e524090

So it works fine in macOS? Maybe my previous testing method was incorrect.

[louis030195]

@Kree0 yes works well, I use it to record .mp3 files & transcribe with OpenAI's whisper 24/7 on my Macbook pro M3 max

[rustdesk]

Good job.

[louis030195]

update:

latest macos release broke this PR (15.0). issue does not happen on 14.5

thread '' panicked at /Users/matthewdi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/objc_id-0.1.1/src/id.rs:52:9: Attempted to construct an Id from a null pointer note: run with RUST_BACKTRACE=1 environment variable to display a backtrace libc++abi: terminating due to uncaught foreign exception zsh: abort ./target/release/screenpipe --debug

[Kree0]

update:

latest macos release broke this PR (15.0). issue does not happen on 14.5

thread '' panicked at /Users/matthewdi/.cargo/registry/src/index.crates.io-6f17d22bba15001f/objc_id-0.1.1/src/id.rs:52:9: Attempted to construct an Id from a null pointer note: run with RUST_BACKTRACE=1 environment variable to display a backtrace libc++abi: terminating due to uncaught foreign exception zsh: abort ./target/release/screenpipe --debug

Can you run it with RUST_BACKTRACE=1 to show the full stack?

[louis030195]

will focus on this PR in upcoming days

[Poordeveloper]

will focus on this PR in upcoming days

Did you get a chance to check the crash?

[louis030195]

https://github.com/SSheldon/rust-objc-id/blob/6527cdf28fef2b155c67c1681f548ecfd794003c/src/id.rs#L52

seems the crash happens here

but on the other hand i see that 15.0 macos is beta right?

so maybe this will be fixed by apple (idk)

will just push to prod my repo with this branch as deps and people that use beta macos wont have access to the feature

[louis030195]

done: https://github.com/louis030195/screen-pipe/commit/ca12e0d8db5d305f41415e1a9340bcb63dc931ac#diff-43858b5b9d45785def40a91685eced0da516c6e6db0038862945361fab28cdde

[louis030195]

fyi there is memory leak

[louis030195]

fyi $150 bounty on fixing this: https://github.com/mediar-ai/screenpipe/issues/183#issuecomment-2310116190

[louis030195]

fyi working on major leaks (there are multiple) related to screencapturekit

https://github.com/mediar-ai/screenpipe/issues/236

[Kree0]

Sorry for the late reply. I have been busy in the past few months. The memory leak seems to have been resolved in mediar-ai/screencapturekit-rs, mediar-ai/screencapturekit-rs. Should I replace the dependencies with them?

[louis030195]

@Kree0 considering we should implement this PR using another lib than screencapturekit-rs

https://github.com/mediar-ai/screenpipe/issues/297

currently i experienced 5-8 users having issues with screencapture kit when macos version below 13, usually seg fault, not sure it's an issue in my implementation or something with macos api

https://github.com/yury/cidre

[kaiwen-wang]

Did any of you try https://crates.io/crates/screen-capture-kit yet?

I also tried ruhear on https://crates.io/crates/ruhear, which seems to cause null pointer problems https://github.com/aizcutei/ruhear/issues/6

[louis030195]

what is the best way to impl this?

@Kree0 @rustdesk @Poordeveloper @kaiwen-wang

$300 bounty to make this work without seg fault or memory leaks

https://github.com/mediar-ai/screenpipe/issues/297#issuecomment-2419992588

[Kree0]

This looks a little difficult. And I don't have a mac device or time to do this in the near future, can see if others have time.

[thewh1teagle]

See https://github.com/yury/cidre/issues/23 This crate seems more stable and makes it easier to add screen capture support to cpal.

[Kree0]

See yury/cidre#23 This crate seems more stable and makes it easier to add screen capture support to cpal.

I‘ll take a look in upcoming days.

[Kree0]

See yury/cidre#23 This crate seems more stable and makes it easier to add screen capture support to cpal.

@louis030195 I replaced it with this, you can see if there is still any problem.

[thewh1teagle]

@Kree0

Tested on macos 14.5 with an M1 chip, and it works! the host/device is visible, and i was able to record wav files using the examples. note that the system prompts for screen recording permissions. however, I think that it's possible to request audio-only recording using screencapturekit.

https://github.com/yury/cidre/issues/23#issuecomment-2453041454

.

[thewh1teagle]

The crate cidre just added option to capture system audio only cidre/examples/core-audio-record/main.rs https://github.com/yury/cidre/issues/23#issuecomment-2498284856

[louis030195]

See yury/cidre#23 This crate seems more stable and makes it easier to add screen capture support to cpal.

@louis030195 I replaced it with this, you can see if there is still any problem.

any chance you can send a PR here with the switch in dependencies in screenpipe-audio/Cargo.toml

https://github.com/mediar-ai/screenpipe/issues/297

so i can test & give you the bounty

[louis030195]

https://giphy.com/gifs/french-week-sDcfxFDozb3bO

[louis030195]

how do i know tehre is no memory leak this time though?

[thewh1teagle]

any chance you can send a PR here with the switch in dependencies in screenpipe-audio/Cargo.toml

I think that It's not ready yet and need a bit more testing how can we declare a vector that holds both regular cpal devices and ScreenCaptureDevice? I tried use:

let devices: Vec<Box<DeviceTrait>>;

It should hold both Device and ScreenCaptureDevice, but I can't call the methods. I must be missing something.

[Kree0]

The crate cidre just added option to capture system audio only cidre/examples/core-audio-record/main.rs yury/cidre#23 (comment)

I think it is not using ScreenCaptureKit but Core Audio. Core Audio seems to have been encapsulated into cpal before, but I don't know if it supports loopback. Currently, a Device of ScreenCaptureKit host captures the output of a display, and it seems that screen recording permission is required.

[thewh1teagle]

but I don't know if it supports loopback.

It support loopback See https://github.com/yury/cidre/blob/main/cidre/examples/core-audio-record/main.rs

https://github.com/yury/cidre/issues/23#issuecomment-2498278269

Maybe with that we won't need another host

Note: I suggest keep this PR changes and maybe create another one with Core Audio or at least keep the commits

[Kree0]

Note: I suggest keep this PR changes and maybe create another one with Core Audio or at least keep the commits

Ok, I'll keep this commit.