madsmtm
Delegate/Protocol methods not getting called
Hi! I've been messing around trying to get ScreenCaptureKit to work and capture a display. I've gotten to the point where the capture succesfully starts, but the methods that are supposed to be handling the output aren't being called externally (but they do trigger when I call them manually).
the two methods that are supposed to be triggered are from:
Here's a link to my demo repo for testing if you'd like to look at the code: https://github.com/hydrobeam/osx_windows_demo/tree/65d11893b7bc815f0d8adeafc41dd30e10a07b4b
I'm not sure if I've defined the external protocols properly or if it's just a case of me not working with ScreenCaptureKit properly. I've checked quite a bit to make sure I don't have basic typos and that the methods I've defined actually correspond to the ones expected by the API, but there's still a chance I might've missed something since it seems to just fail silently.
If you have any suggestions or tips on how to debug this I'd appreciate them!
Thanks for the detailed issue and reproducer.
I am in the process of acquiring a new macbook, so I suspect it will take some time before I will be able to fully diagnose your issue myself.
I do have a suspicion that it's because the protocol is not registered by the runtime, which means that objc2 can't (for now at least) register the class to conform to it, yet ScreenCaptureKit requires the conformance on the runtime level.
Could I get you to give me the output of runtime::Protocol::get("SCStreamDelegate") and runtime::Protocol::get("SCStreamOutput")?
If those return None, then that's likely the problem, and you will need to link in an Objective-C library with the following code (untested, maybe you'll have to make these symbols used somehow to make the C compiler not strip the information completely, I can't quite remember how to do that though):
#include <objc/runtime.h>
#include <ScreenCaptureKit/ScreenCaptureKit.h>
// Define two globals that reference each protocol, to make sure they are
// initialized in the Objective-C runtime.
Protocol* SCStreamOutput_protocol_hack = @protocol(SCStreamOutput);
Protocol* SCStreamDelegate_protocol_hack = @protocol(SCStreamDelegate);
https://github.com/madsmtm/objc2/issues/417 also has roughly the same workaround (again, in case that is indeed the issue)
I've recently added objc2-screen-capture-kit (full version not yet released, but you can tell cargo to fetch it from this repo). Please update your example code to use that, then we're at least sure that the problem isn't with your msg_send!s.
Another issue might be that the application needs to be running, instead of just sleeping on the main thread? Try setting up an application delegate and calling everything inside applicationDidFinishLaunching:, you can find a full example of that here.
Hi, @madsmtm . I just ran into the same problem that SCStreamDelegate and SCStreamOutput not being called.
For example, this is how I define the class:
#[derive(Debug)]
pub(crate) struct StreamOutput {
pub(crate) tx: Sender<Frame>,
}
define_class!(
#[unsafe(super(NSObject))]
#[ivars = StreamOutput]
#[derive(Debug)]
pub(crate) struct VideoStreamOutput;
unsafe impl NSObjectProtocol for VideoStreamOutput {}
#[allow(non_snake_case)]
unsafe impl SCStreamOutput for VideoStreamOutput {
#[unsafe(method(stream:didOutputSampleBuffer:ofType:))]
unsafe fn stream_didOutputSampleBuffer_ofType(
&self,
stream: &SCStream,
sample_buffer: &CMSampleBuffer,
r#type: SCStreamOutputType,
) {
println!("stream_didOutputSampleBuffer_ofType");
// self.ivars().tx.try_send(vec![]).ok();
}
}
);
impl VideoStreamOutput {
pub(crate) fn new(tx: Sender<Frame>) -> Retained<Self> {
unsafe {
let this = Self::alloc().set_ivars(StreamOutput { tx });
msg_send![super(this), init]
}
}
}
This is how I started capturing:
impl CaptureConfig {
/// Spawns a thread to capture the screen and returns a `CaptureDesc` that can be used to control the capture.
pub fn create(self) -> Result<CaptureDesc> {
let (tx, rx) = bounded(self.channel_capacity);
let size = Arc::new(Mutex::new((0, 0)));
let parker = Parker::new();
let unparker = parker.unparker().clone();
let size_guard = size.lock_arc();
let jh = thread::spawn(move || unsafe {
let tx = Box::into_raw(Box::new(tx)); // get a Copy raw ptr for RcBlock
let size_guard = RefCell::new(Some(size_guard));
let thread_parker = Parker::new();
let thread_unparker = thread_parker.unparker().clone();
SCShareableContent::getShareableContentWithCompletionHandler(&RcBlock::new(
move |shareable: *mut SCShareableContent, e: *mut NSError| {
assert!(e.is_null(), "{}", &*e);
let (filter, width, height) = match self.target {
Target::Primary => {
...
}
...
};
let stream_config = SCStreamConfiguration::streamConfigurationWithPreset(
SCStreamConfigurationPreset::CaptureHDRStreamCanonicalDisplay,
);
stream_config.setWidth(width as usize);
stream_config.setHeight(height as usize);
stream_config.setCapturesAudio(false);
stream_config.setCaptureMicrophone(false);
stream_config.setPixelFormat(u32::from_be_bytes(*b"BGRA"));
let stream = SCStream::initWithFilter_configuration_delegate(
SCStream::alloc(),
&filter,
&stream_config,
Some(&ProtocolObject::from_retained(StreamDelegate::new())),
);
stream
.addStreamOutput_type_sampleHandlerQueue_error(
&ProtocolObject::from_retained(VideoStreamOutput::new((*tx).clone())),
SCStreamOutputType::Screen,
None,
)
.unwrap();
stream.startCaptureWithCompletionHandler(Some(&RcBlock::new(
|e: *mut NSError| {
if e.is_null() {
return;
}
error!("{}", &*e);
},
)));
if let Some(mut size_guard) = size_guard.borrow_mut().take() {
*size_guard = (width as u32, height as u32); // Drop size_guard, the main thread continues
}
parker.park();
stream.stopCaptureWithCompletionHandler(None);
thread_unparker.unpark();
},
));
thread_parker.park();
let _ = Box::from_raw(tx);
Ok(())
});
let size = *size.lock(); // Guard in screen capture dropped -> capture started
Ok(CaptureDesc {
jh: AtomicPtr::new(Box::into_raw(Box::new(jh))),
unparker,
rx,
size,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use std::time::Duration;
#[test]
fn capture_test() {
let config = CaptureConfig {
hide: vec![],
channel_capacity: 3,
target: Target::Primary,
};
let capture_desc = config.create().expect("Failed to create capture");
let (width, height) = capture_desc.size();
for _ in 0..10 {
let frame = capture_desc.recv_timeout(Duration::from_secs(4)).unwrap();
assert_eq!(frame.len(), (width * height * 4) as usize);
}
capture_desc.stop();
}
}
The capture started, and actually, if I add stream.addRecordingOutput_error(...), it can also encode and save a media file.
However, just no stream_didOutputSampleBuffer_ofType printed as expected, which means the delegate wasn't called.
Any help is so kind!!! Full version code at #768
