Idea: Add CoreML / Apple Neural Engine backend

[janekb04]

I don't know anything about CoreML other than the fact that the Neural Engine is very well optimized on Apple's devices. I'm writing this as an idea. Would it be possible to introduce this backend?

I know that Apple has its own CoreML Transformer implementations, and that similar support has been recently added to whisper.cpp, where it brought a significant performance improvement.

[guillaumekln]

The issue is that CoreML is not really a backend. It is a high-level framework to execute models so it does fit nicely in our current design.

Regarding the Apple Neural Engine, AFAIK there is currently no API to use it directly. It can only be used via CoreML.

https://github.com/hollance/neural-engine

[janekb04]

It is unfortunate that the Neural Engine cannot be programmed directly. I mentioned CoreML, as this could potentially significantly improve performance of CTranslate2, at least on the newer Apple hardware.