[WIP] Add AVFoundation Loader and MonoLoader.

[Ivorforce]

What

I copied and adapted the default AudioLoader and MonoLoader classes to load using AVFoundation rather than ffmpeg. These are separate loaders that can be run natively on apple products (macOS, iOS, ipadOS), and nowhere else.

This PR is a proof of concept. Let me know if there is any interest in pursuing this endeavour further.

Why

AVFoundation is powerful and fast. For mac users or apps, there is no need for an additional dependency to ffmpeg.

ToDos

I've very shoddily implemented and tested these loaders. As far as my tests go right now, they produce identical results to the ffmpeg loaders.

There's a bunch of things not right with the code, as one can easily see. Some I've annotated, but here's a quick list ordered by size / importance:

• [ ] this PR necessarily contains an objective-c++ file. I am not sure how easy it is to get this to compile using current configurations
• [ ] this PR necessarily depends on AVFoundation. Somehow, this needs to be detected and compiled against properly
• [ ] files are not listed in waf to compile
• [ ] loaders are not registered in the factory
• [x] deinit is not implemented properly
• [ ] code is not cleaned up
• [x] indentation problems
• [ ] close to no documentation
• [ ] md5 computation is not implemented (if needed)

To be honest, I don't know how this project is set up exactly. I'm additionally not familiar with waf or the build setup, and while I did get it to compile and run on my end, the setup might be suboptimal. I'd be grateful for any assistance in this PR.

If Objective-C + AVFoundation compiling support is too far out, which it might as well be, this PR could also be modified to be a toolbox. If loaders were implemented as templates, and a sample implementation of the AVFoundationAudioFile class was provided somewhere, it would reduce overhead to any users simply adapting this interface in their existing objective-c++ projects greatly.

[Ivorforce]

Update: I fixed some memory management and whitespace issues.

Anyone looking to copy this code should note that in the current release, AVMonoLoader's .disconnect statements will cause a crash and should comment them out.

[rahul140490]

Hi, your work looks great. Thanks for that. But, would you like to describe how can I use this in my iOS project for calculating MFCC values of a local audio file?

[Ivorforce]

Hi @rahul140490, this merge request merely adds a file loader that uses AVFoundation for IO. You'd be able to use Essentia in your iOS project even without it (using ffmpeg instead).

If you need reference code, I've got a project up that interfaces between Essentia and Objective-C here. The project file also automates the setup of Essentia, but in general, I'll refer to Essentia's own install docs for these things.

Cheers

[dbogdanov]

Hi @Ivorforce, thank you for the PR, this is something that is definitely interesting to support in Essentia. So as I understand you did not use waf to build? It suppose is possible to have Objective-C++ files and use our current build system but I haven't looked at it. Our build script creates a list of algorithms to include to AlgorithmFactory here.

One thing to think about is whether we should re-use the same algorithm names (that is AudioLoader instead of AVAudioLoader) for consistency as there are few algorithms and many code examples relying on AudioLoader/MonoLoader (such as MusicExtractor, FreesoundExtractor, many of our Python examples).

For example, in the build script we have a flag to select the FFT type, which will then use a specific algorithm class for the FFT implementation and ignore other algorithms. We have FFTC, FFTA and FFTK all having the same name FFT in the AlgorithmFactory.

[Ivorforce]

@dbogdanov Correct, I did not use Waf to build the project. Basically, I wrote the files for my own project (where they run fine) and figured these could be a useful addition to Essentia itself.

Coming from OOP, I would have suggested to have at least MonoLoader implemented as a template to some AudioLoader. I guess compiling against a specific kind of AudioLoader would be an alternative solution - while imo it would be a bit restrictive for a user to have to decide for one of the loaders at compile time, I don't feel too strongly on this. Switching in a running application is fortunately not a use-case for me.

[dbogdanov]

We'll keep this PR pending as we'll need to look at updating the waf build script accordingly.