hollance
Detecting Supported Layers
Hey there! This repo is super amazing, thanks for putting all these findings together.
Was reading this sub-section (link below) on which layers are unsupported, and I had an idea on how to programmatically identify them.
https://github.com/hollance/neural-engine/blob/master/docs/unsupported-layers.md
This part here: "S → U → S → U → S → U" about swapping supported/unsupported layers made me wonder. In theory, if we take a layer X from a CoreML model (or just some CoreML op) we have that we do not yet know can run on the ANE, and we make a dummy model that is built solely of that layer, i.e. X -> X -> X ... and we set the compute unit to be CPU/ANE only, this would encourage the Neural Engine to run this model on 1 compute unit since that would be efficient. So, in theory, if you set the compute unit to be CPU && ANE and you see that the layer runs on the ANE only, then you will have identified that this op is ANE compatible? 🧐 could even record stats as well.
I'd like to test this theory but wanted to run this by you. Thinking this could be a way to, given a model, programmatically piecemeal individual layers, build a simple repeated layer model, and produce a chart of whether the layer is CPU/GPU/ANE supported (maybe even with statistics). Maybe even a chart that can be publically available of ops and their supported compute units (since something like that does not exist today to my knowledge).
Would help with identifying areas where a layer could be swapped out/modified to encourage running the model more efficiently on the ANE.
Any thoughts would be appreciated! 😊
@hollance thanks for writing those CoreML survival guides, I find them to be an invaluable supplement for apple's own documentation. I've started working on this, calling it "anetools" 👊
Using your section of "Model Surgery" I was able to successfully isolate the first layer programmatically in a small Neural Network! Cool thing is I could run the performance tab on it too.
Going to work on making this generically possible for all layers in the model, which will require a bit more work than just input/output name matching (shapes and datatypes too which will be a bit difficult).
Off the top of your head, do you know how I could programmatically map the layer.output shape (which is not as straightforward as using layer.output.shape, probably more complex) to the equivalent model.output_description.type?
Honestly, it's been too long since I did anything with Core ML so I don't know off the top of my head.
No worries! I think I found it deep within some proto objects. Gonna have to add in dynamic data building per data type per input (there's only a handful of FeatureType(s) so not too bad).
@hollance was this feature in XCode around when you were working on CoreML? (Performance tab, you can see it in the link).
I saw the apple discussion/article was from WWDC22' but not sure if it was around back then.
It somehow detects the compute unit per layer, and also tells you whether a layer is compatible with a given compute unit. It sometimes doesn't work but when it does it is pretty nice.
Seems like Apple already implemented a "supported layers" feature kinda 😂
https://developer.apple.com/videos/play/wwdc2022/10027/
Unfortunately that feature was not available when I was working with Core ML. Would have been useful at the time. :-)
Yeah I realized this does most of what I'm trying to research 😆 although it does on occasion fail for ML Program models it seems. If you like I could make some README updates to this project in reflection of the features in XCode for detecting ANE layer support, and whether it gets picked up.