macos audio output Display with screencapturekit crash sometimes - (switch to cidre for macos audio output)

[louis030195]

https://github.com/yury/cidre

i think there is another repo forgot link

basically we rely on https://github.com/mediar-ai/screencapturekit-rs through a hack in https://github.com/mediar-ai/cpal

which fixes ton of memory leaks but creates seg fault some times i think

[linear[bot]]

MED-89 switch to cidre for macos audio output

[louis030195]

also sometimes in current impl:

2024-09-10T17:48:37.242815Z ERROR screenpipe_audio::core: An error occurred on the audio stream: A backend-specific error has occurred: error occurred in screencapturekit stream

i think this is when computer sleeps #361

[louis030195]

https://github.com/yury/cidre/blob/main/cidre/examples/sc-record/main.rs

[louis030195]

/bounty 300

ideally seamless integration with cpal, please first suggest how you will implement this before working on it (cidre, screencapture-kit etc.)

PS: likely will just do this myself

[algora-pbc[bot]]

💎 $300 bounty • Screenpi.pe

Steps to solve:

1. Start working: Comment /attempt #297 with your implementation plan
2. Submit work: Create a pull request including /claim #297 in the PR body to claim the bounty
3. Receive payment: 100% of the bounty is received 2-5 days post-reward. Make sure you are eligible for payouts

Thank you for contributing to mediar-ai/screenpipe!

Add a bounty • Share on socials

[kaiwen-wang]

For my purposes I decided to just build a swift app

Try reaching out to the authors of those who made rust bindings

[twilwa]

I'm pretty deep in the weeds trying to solve the linux audio issue on my branch and attempt to resolve #450 , and ended up stumbling on https://github.com/CapSoftware/scap as an alternative to screencapture-kit. or rather, what claims to be a more modern/simplified API to access it. Might be worth a look, happy to try and implement and see if it works out (although I won't be on mac, so I can't test ScreenCaptureKit specifically.)

For what it's worth, I'd prefer to move away from cpal for linux audio as well -- they only support ALSA and JACK backends, ALSA appears to have some trouble with default devices https://github.com/RustAudio/cpal/issues/226 and JACK requires a submodule that I've managed to install, but still getting the same ALSA error when trying to configure the JACK server.

My solve for linux audio, I think, will probably be to try and just use ffmpeg to capture the audio in addition to the video. It looks like that could be an option for Mac as well, seems like there's a stable build for macos on the ffmpeg site.

[louis030195]

I'm pretty deep in the weeds trying to solve the linux audio issue on my branch and attempt to resolve #450 , and ended up stumbling on https://github.com/CapSoftware/scap as an alternative to screencapture-kit. or rather, what claims to be a more modern/simplified API to access it. Might be worth a look, happy to try and implement and see if it works out (although I won't be on mac, so I can't test ScreenCaptureKit specifically.)

For what it's worth, I'd prefer to move away from cpal for linux audio as well -- they only support ALSA and JACK backends, ALSA appears to have some trouble with default devices RustAudio/cpal#226 and JACK requires a submodule that I've managed to install, but still getting the same ALSA error when trying to configure the JACK server.

My solve for linux audio, I think, will probably be to try and just use ffmpeg to capture the audio in addition to the video. It looks like that could be an option for Mac as well, seems like there's a stable build for macos on the ffmpeg site.

yes i also dislike cpal, not really maintained, not accepting PRs etc.

probably will end up implementing something ourselves, ffmpeg seems a solid option, wonder if we can capture input & output (system audio)

[twilwa]

we can -- i'm at the point locally where i can capture audio without errors, just need to figure out how to write it to a file to verify

On Sat, Oct 19, 2024 at 10:49 AM Louis Beaumont @.***> wrote:

I'm pretty deep in the weeds trying to solve the linux audio issue on my branch and attempt to resolve #450 https://github.com/mediar-ai/screenpipe/issues/450 , and ended up stumbling on https://github.com/CapSoftware/scap as an alternative to screencapture-kit. or rather, what claims to be a more modern/simplified API to access it. Might be worth a look, happy to try and implement and see if it works out (although I won't be on mac, so I can't test ScreenCaptureKit specifically.)

For what it's worth, I'd prefer to move away from cpal for linux audio as well -- they only support ALSA and JACK backends, ALSA appears to have some trouble with default devices RustAudio/cpal#226 https://github.com/RustAudio/cpal/issues/226 and JACK requires a submodule that I've managed to install, but still getting the same ALSA error when trying to configure the JACK server.

My solve for linux audio, I think, will probably be to try and just use ffmpeg to capture the audio in addition to the video. It looks like that could be an option for Mac as well, seems like there's a stable build for macos on the ffmpeg site.

yes i also dislike cpal, not really maintained, not accepting PRs etc.

probably will end up implementing something ourselves, ffmpeg seems a solid option, wonder if we can capture input & output (system audio)

— Reply to this email directly, view it on GitHub https://github.com/mediar-ai/screenpipe/issues/297#issuecomment-2424106603, or unsubscribe https://github.com/notifications/unsubscribe-auth/A2PEDCT4JVFJZOARN2JXQRDZ4KLRRAVCNFSM6AAAAABN5PK7ZOVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMRUGEYDMNRQGM . You are receiving this because you commented.Message ID: @.***>

[thewh1teagle]

Did you solved the issues with screencapturekit?

[louis030195]

@thewh1teagle

Did you solved the issues with screencapturekit?

no, so i think there are two issues:

1. "An error occurred on the audio stream: A backend-specific error has occurred: error occurred in screencapturekit stream" -> probably just adding a retry somewhere would fix
2. segmentation fault / bus error -> seems only happening on specific versions of macos https://github.com/mediar-ai/screenpipe/pull/490 @ologbonowiwi

this fork fix the mountain of memory leaks that were in the main branch https://github.com/mediar-ai/screencapturekit-rs

but not sure this is my changes that added the seg fault

another workaround would be a rust wrapper around https://github.com/ExistentialAudio/BlackHole to automatically create a loopback with speakers and make it an audio output in cpal and avoid using screencapturekit

[thewh1teagle]

another workaround would be a rust wrapper around https://github.com/ExistentialAudio/BlackHole to automatically create a loopback with speakers and make it an audio output in cpal and avoid using screencapturekit

That will requires the user to install blackhole and configure it, i wouldn't use it

[louis030195]

There is a C library I think

[louis030195]

let me run this for a bit then i'll give you the bounty @kree0

[shenshouer]

rust-analyzer on vscode report error about cidre @louis030195

[Error - 11:10:08 AM] Server process exited with code 0.
2024-11-27T11:10:39.418138+08:00 ERROR FetchBuildDataError: error: failed to run custom build command for `cidre v0.4.0 (https://github.com/yury/cidre.git?rev=f05c428#f05c4288)`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `/Users/sope/workspaces/ai/screenpipe/target/debug/build/cidre-e7ce18c2ba1af0ba/build-script-build` (exit status: 101)
  --- stdout
  cargo:rustc-link-lib=clang_rt.osx
  cargo:rustc-link-search=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/lib/darwin
  cargo:rustc-link-search=framework=/System/Library/PrivateFrameworks
  cargo:rustc-link-search=framework=/Library/Apple/System/Library/PrivateFrameworks
  cargo:rustc-link-search=native=/Users/sope/workspaces/ai/screenpipe/target/debug/build/cidre-33c596282065fdc5/out/Debug

  --- stderr
  2024-11-27 11:10:39.346 xcodebuild[71114:10644544] [MT] DVTPlugInLoading: Failed to load code for plug-in com.apple.dt.IDESimulatorFoundation (/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework), error = Error Domain=NSCocoaErrorDomain Code=3588 "dlopen(/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, 0x0109): Library not loaded: /Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads
    Referenced from: <A02FEDF7-46B7-3643-B691-743821CCE736> /Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/Versions/A/IDESimulatorFoundation
    Reason: tried: '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file)" UserInfo={NSLocalizedFailureReason=The bundle couldn’t be loaded., NSLocalizedRecoverySuggestion=Try reinstalling the bundle., NSFilePath=/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, NSDebugDescription=dlopen(/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, 0x0109): Library not loaded: /Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads
    Referenced from: <A02FEDF7-46B7-3643-B691-743821CCE736> /Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/Versions/A/IDESimulatorFoundation
    Reason: tried: '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), NSBundlePath=/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework, NSLocalizedDescription=The bundle “IDESimulatorFoundation” couldn’t be loaded.}, dyldError = dlopen(/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, 0x0000): Library not loaded: /Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads
    Referenced from: <A02FEDF7-46B7-3643-B691-743821CCE736> /Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/Versions/A/IDESimulatorFoundation
    Reason: tried: '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file)

  A required plugin failed to load. Please ensure system content is up-to-date — try running 'xcodebuild -runFirstLaunch'.
  thread 'main' panicked at /Users/sope/.cargo/git/checkouts/cidre-e660120dbd7e907a/f05c428/cidre/build.rs:145:5:
  assertion failed: status.success()
  stack backtrace:
     0: rust_begin_unwind
               at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
     1: core::panicking::panic_fmt
               at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
     2: core::panicking::panic
               at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:148:5
     3: build_script_build::xc_build
     4: build_script_build::main
     5: core::ops::function::FnOnce::call_once
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.


2024-11-27T11:10:47.010476+08:00 ERROR Flycheck failed to run the following command: CommandHandle { program: "/Users/sope/.cargo/bin/cargo", arguments: ["check", "--workspace", "--message-format=json-diagnostic-rendered-ansi", "--manifest-path", "/Users/sope/workspaces/ai/screenpipe/Cargo.toml", "--keep-going", "--all-targets", "--features", "metal pipes"], current_dir: Some("/Users/sope/workspaces/ai/screenpipe") }, error=Cargo watcher failed, the command produced no valid metadata (exit code: ExitStatus(unix_wait_status(25856))):
   Compiling cidre v0.4.0 (https://github.com/yury/cidre.git?rev=f05c428#f05c4288)
error: failed to run custom build command for `cidre v0.4.0 (https://github.com/yury/cidre.git?rev=f05c428#f05c4288)`
note: To improve backtraces for build dependencies, set the CARGO_PROFILE_DEV_BUILD_OVERRIDE_DEBUG=true environment variable to enable debug information generation.

Caused by:
  process didn't exit successfully: `/Users/sope/workspaces/ai/screenpipe/target/debug/build/cidre-e7ce18c2ba1af0ba/build-script-build` (exit status: 101)
  --- stdout
  cargo:rustc-link-lib=clang_rt.osx
  cargo:rustc-link-search=/Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/lib/clang/16/lib/darwin
  cargo:rustc-link-search=framework=/System/Library/PrivateFrameworks
  cargo:rustc-link-search=framework=/Library/Apple/System/Library/PrivateFrameworks
  cargo:rustc-link-search=native=/Users/sope/workspaces/ai/screenpipe/target/debug/build/cidre-33c596282065fdc5/out/Debug

  --- stderr
  2024-11-27 11:10:46.901 xcodebuild[71546:10646028] [MT] DVTPlugInLoading: Failed to load code for plug-in com.apple.dt.IDESimulatorFoundation (/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework), error = Error Domain=NSCocoaErrorDomain Code=3588 "dlopen(/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, 0x0109): Library not loaded: /Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads
    Referenced from: <A02FEDF7-46B7-3643-B691-743821CCE736> /Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/Versions/A/IDESimulatorFoundation
    Reason: tried: '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file)" UserInfo={NSLocalizedFailureReason=The bundle couldn’t be loaded., NSLocalizedRecoverySuggestion=Try reinstalling the bundle., NSFilePath=/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, NSDebugDescription=dlopen(/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, 0x0109): Library not loaded: /Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads
    Referenced from: <A02FEDF7-46B7-3643-B691-743821CCE736> /Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/Versions/A/IDESimulatorFoundation
    Reason: tried: '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), NSBundlePath=/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework, NSLocalizedDescription=The bundle “IDESimulatorFoundation” couldn’t be loaded.}, dyldError = dlopen(/Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/IDESimulatorFoundation, 0x0000): Library not loaded: /Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads
    Referenced from: <A02FEDF7-46B7-3643-B691-743821CCE736> /Applications/Xcode.app/Contents/Frameworks/IDESimulatorFoundation.framework/Versions/A/IDESimulatorFoundation
    Reason: tried: '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/System/Volumes/Preboot/Cryptexes/OS/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file), '/Library/Developer/PrivateFrameworks/DVTDownloads.framework/Versions/A/DVTDownloads' (no such file)

  A required plugin failed to load. Please ensure system content is up-to-date — try running 'xcodebuild -runFirstLaunch'.
  thread 'main' panicked at /Users/sope/.cargo/git/checkouts/cidre-e660120dbd7e907a/f05c428/cidre/build.rs:145:5:
  assertion failed: status.success()
  stack backtrace:
     0: rust_begin_unwind
               at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:662:5
     1: core::panicking::panic_fmt
               at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:74:14
     2: core::panicking::panic
               at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/panicking.rs:148:5
     3: build_script_build::xc_build
     4: build_script_build::main
     5: core::ops::function::FnOnce::call_once
  note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

[louis030195]

@shenshouer what's that? what did you run?

do you have xcode stuff installed and updated?

[shenshouer]

@shenshouer what's that? what did you run?

do you have xcode stuff installed and updated? @louis030195

Before attempting to open Screenpipe using VSCode, I updated Xcode and agreed to the LICENSE in the command line. When I tried to open the Screenpipe project in VSCode, the issue mentioned above occurred. However, when I reopened the updated Xcode, it prompted me to choose the necessary SDK for download. I did not select any iOS SDKs, keeping only a default, unselectable macOS SDK. I clicked through to open Xcode, then closed it. Upon trying to open the Screenpipe project in VSCode again, the issue had disappeared.

[Kree0]

let me run this for a bit then i'll give you the bounty @Kree0

Ok, hope it works fine. I'm looking forward to the bounty. :satisfied:

[Tushar4059x]

is this issue still open?