No kit found for matmul on 0.21.8

[andrenatal]

I tried to run multiple transformers models using the examples in the repo on 0.21.8 and always got errors missing the MatMul ops. Examples:

with sentence transformers

Error: running pass codegen

Caused by:
    0: codegen node #420 "/encoder/layer.0/attention/self/query/MatMul" EinSum
    1: Translating to OptMatMul
    2: No kit found for matmul 384,384,F32 -0.14861962, 0.035969656, -0.059760537, 0.015408975, 0.016326722, 0.09061663, -0.18811563, 0.13049862, -0.04497628, -0.06142004, 0.017121239, -0.032135025... • batch_size,sequence_length,384,F32

with distilbert:

Failed to load model: running pass codegen

Caused by:
    0: codegen node #416 "/transformer/layer.0/attention/q_lin/MatMul" EinSum
    1: Translating to OptMatMul
    2: No kit found for matmul 768,768,F32 -0.0024362782, 0.007969285, -0.004340109, -0.046699107, -0.035658833, -0.020134414, -0.018087152, -0.018782265, 0.057385396, 0.010039896, -0.000028961475, -0.04873895... • batch_size,sequence_length,768,F32

with albert:

Failed to load model: running pass codegen

Caused by:
    0: codegen node #690 "/albert/encoder/embedding_hidden_mapping_in/MatMul" EinSum
    1: Translating to OptMatMul
    2: No kit found for matmul 128,768,F3

After reverting to 0.19.8, it started to work again

[kali]

Hey, sorry for that, the kit abstraction is a pretty big change in the matrix multiplication code. What target are you running on ?

[andrenatal]

This is native runtime on Apple M2.

[kali]

Are we really running native arm64 here ? not emulated x86_64 over an arm64 system ?

Can you run a tract kernels on 0.21.8 and copy paste the output ?

[kali]

May relate to the re-opening of #1022 .

[andrenatal]

Hi @kali . Yes, that was the root cause:

Error: running pass codegen

Caused by:
    0: codegen node #420 "/encoder/layer.0/attention/self/query/MatMul" EinSum
    1: Translating to OptMatMul
    2: No kit found for matmul 384,384,F32 -0.14861962, 0.035969656, -0.059760537, 0.015408975, 0.016326722, 0.09061663, -0.18811563, 0.13049862, -0.04497628, -0.06142004, 0.017121239, -0.032135025... • batch_size,sequence_length,384,F32
anatal@MacBookPro sentence_transformers % file target/release/sentence-transformers
target/release/sentence-transformers: Mach-O 64-bit executable x86_64

I managed to run with 0.21.8 on my x86_64 Ubuntu and when building targeting --target aarch64-apple-darwin on M2:

anatal@MacBookPro sentence_transformers % file target/aarch64-apple-darwin/release/sentence-transformers
target/aarch64-apple-darwin/release/sentence-transformers: Mach-O 64-bit executable arm64

Just out of curiosity: what's the reason that made the emulated x86_64 build run on arm64 on 0.19 but not with 0.21?

Thanks for helping out.

[kali]

Just wanted to say, it is a bit weird that your mac is not on arm64 by default.

As for x86 emulated on an arm64 mac not working, it's an interesting question, i definitiely want to have a look.

[andrenatal]

Just wanted to say, it is a bit weird that your mac is not on arm64 by default.

I know, I had this problem before as well in a different project. I think the issue is that whenever I get a new laptop, I restore it using time machine and who knows what comes along, and sometimes issues like this, comes back to haunt me.