✨ Allow audio processing similar to frame processing

[manuelbieh]

What feature or enhancement are you suggesting?

I'd love to be able to analyze the audio channel during the recording of a video like you can do it with the video channel with frame processors. Not sure if that's not possible already but from what I understood by reading the docs, frame processing plugins only receive an image buffer with no audio included. Would be great to be able to analyze the audio channel too. In the particular case I have in mind I would need to be able to check the audio channel of a video for short volume peaks.

What Platforms whould this feature/enhancement affect?

iOS, Android

Alternatives/Workarounds

I don't think there's a workaround to that yet. I could probably record audio and video separately and analyze the audio data with a library other than vision camera in parallel and later merge audio and video back together, not sure if that's possible though. Luckily I don't need that feature yet but I probably will in the future. I'd be happy to help you here, but I have never worked with Swift or Objective-C before so I don't know if I could really help you at this point. I could however help you with the RN/JS part. I have also implemented something like that in a web app not long ago so I could provide you with the code to give you an idea of how that could work.

Additional information

• [X] I agree to follow this project's Code of Conduct
• [X] I searched for similar feature requests in this repository and found none.

[mrousavy]

Hey Manuel, thanks for the idea! I think I already posted this idea somewhere, maybe GH discussions 🤔

It should definitely be possible and I've actually done almost this exact thing for Expo before (see: https://github.com/expo/expo/pull/13516). It will be a bit tricky to

1. integrate with VisionCamera (especially because of the parallel Frame Processor runtime, maybe create another one?)
2. integrate with JS, pre-alloc buffers etc.
3. create another plugin API
4. normalize all of the data to an easy to use format (PCM Buffers = raw byte arrays, one array per channel)

It's possible, and I'd love to work on this, but I don't have the free time right now to do that 😄

[adam-a]

Hi @mrousavy hope all is well! Was this ever added in?

[mrousavy]

Hey @adam-a! Not yet, no. Been working on lots of other things for V3 in the meantime.

[lumore]

Hey @manuelbieh! Sorry to tag you, but did you manage to find a workaround?