mistral.rs
mistral.rs copied to clipboard
EricLBuehler
Fix Metal python package running on CPU
Refs #555. @KaQuMiQ I added some debug statements to get a better picture of what's going on. Can you please install from source:
(assuming you have Rust installed, which I think you do if you can run the PyPI package):
git clone https://github.com/EricLBuehler/mistral.rs.git
cd mistral.rs
git pull
git switch metal_on_cpu
< Activate a Python env now to install into >
pip install maturin[patchelf]
maturin develop -r --features metal
If you could try this out that would be great.
Code Metrics Report
=============================================================================== Language Files Lines Code Comments Blanks =============================================================================== C Header 2 35 28 0 7 Dockerfile 1 34 25 0 9 Happy 1 442 369 0 73 JSON 11 102 101 0 1 Python 41 1586 1368 46 172 TOML 19 558 492 11 55 ------------------------------------------------------------------------------- Jupyter Notebooks 2 0 0 0 0 |- Markdown 2 77 32 31 14 |- Python 2 196 169 1 26 (Total) 273 201 32 40 ------------------------------------------------------------------------------- Markdown 24 1817 0 1371 446 |- BASH 5 101 98 0 3 |- JSON 1 12 12 0 0 |- Python 5 92 82 0 10 |- Rust 6 407 364 19 24 |- TOML 2 75 63 0 12 (Total) 2504 619 1390 495 ------------------------------------------------------------------------------- Rust 169 54555 49536 970 4049 |- Markdown 91 848 13 785 50 (Total) 55403 49549 1755 4099 =============================================================================== Total 271 59129 51919 2398 4812 ===============================================================================
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}
I have failed to build the package that way. I have read your release script and modified manually Cargo.toml inside mistralrs-pyo3 folder and built there without installing patchelf (failed to install that one) - it seems to work now with metal acceleration. Here is build log:
> maturin develop -r --features metal
⚠️ Warning: You specified maturin ==1.4 in pyproject.toml under `build-system.requires`, but the current maturin version is 1.7.0
🔗 Found pyo3 bindings
🐍 Found CPython 3.12 at /Users/miquido/Projects/mistral.rs/.venv/bin/python
📡 Using build options profile from pyproject.toml
Compiling pyo3-build-config v0.22.1
Compiling serde_json v1.0.120
Compiling tracing v0.1.40
Compiling tower v0.4.13
Compiling mistralrs-core v0.2.0 (/Users/miquido/Projects/mistral.rs/mistralrs-core)
Compiling h2 v0.4.5
Compiling candle-metal-kernels v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=c967be9#c967be9c)
Compiling tracing-subscriber v0.3.18
Compiling pyo3-ffi v0.22.1
Compiling pyo3-macros-backend v0.22.1
Compiling pyo3 v0.22.1
Compiling mistralrs-pyo3 v0.2.0 (/Users/miquido/Projects/mistral.rs/mistralrs-pyo3)
Compiling safetensors v0.4.3
Compiling ureq v2.10.0
Compiling hyper v1.4.1
Compiling askama v0.12.1
Compiling candle-core v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=c967be9#c967be9c)
Compiling hyper-util v0.1.6
Compiling plotly_kaleido v0.9.0
Compiling pyo3-macros v0.22.1
Compiling hyper-tls v0.6.0
Compiling reqwest v0.12.5
Compiling plotly v0.9.0
Compiling hf-hub v0.3.2
Compiling tokenizers v0.19.1
Compiling candle-nn v0.6.0 (https://github.com/EricLBuehler/candle.git?rev=c967be9#c967be9c)
Compiling mistralrs-vision v0.2.0 (/Users/miquido/Projects/mistral.rs/mistralrs-vision)
Finished `release` profile [optimized] target(s) in 45.80s
📖 Found type stub file at mistralrs.pyi
📦 Built wheel for CPython 3.12 to /var/folders/9x/1vm9c8gj3jb_mm3b8yg2rrvw0000gn/T/.tmpeadSgs/mistralrs-0.2.0-cp312-cp312-macosx_11_0_arm64.whl
✏️ Setting installed package as editable
🛠 Installed mistralrs-0.2.0
Here is my modifierd Cargo.toml:
[package]
name = "mistralrs-pyo3"
authors = ["Eric Buehler"]
version.workspace = true
edition.workspace = true
description.workspace = true
repository.workspace = true
keywords.workspace = true
categories.workspace = true
license.workspace = true
homepage.workspace = true
[lib]
name = "mistralrs"
crate-type = ["cdylib"]
doc = false
[dependencies]
pyo3.workspace = true
mistralrs-core = { version = "0.2.0", path = "../mistralrs-core", features = ["pyo3_macros", "metal"] }
serde.workspace = true
serde_json.workspace = true
candle-core = { git = "https://github.com/EricLBuehler/candle.git", version = "0.6.0", rev = "c967be9", features=["metal"] }
indexmap.workspace = true
accelerate-src = { workspace = true, optional = true }
intel-mkl-src = { workspace = true, optional = true }
either.workspace = true
futures.workspace = true
tokio.workspace = true
image.workspace = true
reqwest.workspace = true
base64.workspace = true
[build-dependencies]
pyo3-build-config = "0.22"
[features]
cuda = ["candle-core/cuda", "mistralrs-core/cuda"]
cudnn = ["candle-core/cudnn", "mistralrs-core/cudnn"]
metal = ["candle-core/metal", "mistralrs-core/metal"]
flash-attn = ["cuda", "mistralrs-core/flash-attn"]
accelerate = ["mistralrs-core/accelerate"]
mkl = ["mistralrs-core/mkl"]
Updated two lines comparing to the original:
mistralrs-core = { version = "0.2.0", path = "../mistralrs-core", features = ["pyo3_macros", "metal"] }
and
candle-core = { git = "https://github.com/EricLBuehler/candle.git", version = "0.6.0", rev = "c967be9", features=["metal"] }
