Capture frame output of applied effect

[jitensndk]

Hello,

How can i get captureOutput ( CMSampleBuffer ) frame after effect is applied, i want to process frame for face detection. i tried to expose this method via delegate and it's working but camera feed started lagging after use of delegation

[Silence-GitHub]

We do not get and process CMSampleBuffer directly. For face detection, we can use metadataObjectDelegate

class MyViewController: UIViewController {
    override func viewDidLoad() {
        ...
        camera = ...
        camera.addMetadataOutput(with: [.face])
        camera.metadataObjectDelegate = self
        ...
    }
}

extension MyViewController: BBMetalCameraMetadataObjectDelegate {
    func camera(_ camera: BBMetalCamera, didOutput metadataObjects: [AVMetadataObject]) {
        guard let first = metadataObjects.first else { return }
        DispatchQueue.main.async { [weak self] in
            guard let self = self else { return }
            self.handleFaceBounds(first.bounds)
        }
    }
}

[Silence-GitHub]

@jitensndk Hello, I reconsider the issue. The metadataObjectDelegate is not powerful enough. Could you please tell me how do you detect face (Core Image, Core ML or other)? Can you detect face with metal texture? Maybe we can add a new feature for face detection.

[jitensndk]

Hello @Silence-GitHub i detect face using vision framework. as we get cvsamplebuffer from didoutput of avfoundation i pass buffer to vision framework and it returns no of faces found in one frame. do let me know if i could clear your question

[blurtime]

@jitensndk Has there been any progress on this? I also looked into Vision and might try to add face detection (or something like a face tracker zooming in on a face with the zoom filter). If things work out well, I'll create some pull requests soon.

[blurtime]

I managed to implement the most important step. I can feed the frames into Vision and it successfully recognizes how many faces are in a frame. @Silence-GitHub You seemed to be interested in this. I'll create a pull request ASAP.

[jitensndk]

Hello @blurtime , @Silence-GitHub face detection is already working fine using vision framework, all i wanted to apply some image pre processing for face recognition, so i used some filters like brightness and exposure to get face detected in low light condition. but it seemed feeding image frames in both vision framework and bbmetalimage framework at same time turned out to be legging experience. if you want i can provide sample app for you to review.

[blurtime]

@jitensndk If you still have the code and think it's working better/more smoothly than mine in this pull request #67, I'd be more than happy to see it. I thought it didn't work because you mentioned that it wasn't detecting faces ("it returns no of faces found in one frame").

[jitensndk]

@blurtime sorry i could not explain and use "it returns no of faces found in one frame" instead of "it returns number of faces found in one frame". sorry i am not native english speaker.

[Silence-GitHub]

Please try version 1.1.7. We can get original video sample buffer in preprocessVideo block.

camera.preprocessVideo = { sampleBuffer in
    // Do something with original video sample buffer
}

[jitensndk]

@Silence-GitHub let me know if i can be any help as i exploring face application for biometric.

[floriangbh]

Hello @Silence-GitHub, it possible to have the same process with BBMetalUISource ?

I would like to have a captureOutput ( CMSampleBuffer ) frame after effect is applied, but I don't find a way. preprocessVideo is not present on BBMetalUISource.

Thanks

[Silence-GitHub]

@floriangbh No. Now this library provides MTLTexture, not CMSampleBuffer. Camera and video source provide original CMSampleBuffer because of AVFoundation.