Falcon example seems broken (on metal)

[jorgeantonio21]

I am trying to run falcon locally on my machine, on main branch, through:

cargo run --release --features metal --example falcon -- --prompt "write a hello world rust program"

which fails with error

Error: Metal error matmul doesn't support BF16

[LaurentMazare]

That's a known limitation of the metal backend at the moment. @ivarflakstad has been looking at getting this to work but not sure what the current state is.

[ivarflakstad]

I have this branch with working bfloat matmul. I'm testing running falcon on it now (downloading)

It is based on work I've done here which is not ready to be merged.

[ivarflakstad]

If you have enough RAM you should be able to run Falcon on the candle branch I mentioned above. Here I am running Mamba (130m) with bf16: