ndarray not compiling on Mac custom silicon

[M1KE-TAYL0R]

I am trying to run some simulations that I wrote on an old machine on my new MacBook Pro with an M1 Pro chip. I have already installed openblas via brew; however when trying to compile my code, I received the following error:

% cargo build                   

   Compiling openblas-src v0.10.4
error: failed to run custom build command for `openblas-src v0.10.4`

Caused by:
  process didn't exit successfully: `/Users/mike/Documents/Photonic_Truncation/target/debug/build/openblas-src-26a246e21ed617e6/build-script-build` (exit status: 101)
  --- stdout
  Running: `"make" "libs" "netlib" "shared" "BINARY=64" "YES_CBLAS=1" "NO_LAPACKE=1" "-j10"`

After doing some experimenting, this seems to be an endemic problem when trying to build on Apple's chips.

Is there some work around for this?

[sstadick]

@termoshtt / @jturner314 has anyone figured out how to get this working? Or is it left unsupported for now?

[sstadick]

Using the openblas-system backend, installing openblas via homebrew, and adding the following to my build.rs seems to have worked.

     println!("cargo:rustc-link-search=/opt/homebrew/opt/openblas/lib");

[jturner314]

That's a good workaround. The build error for the openblas-static feature looks like an issue with openblas-src, not ndarray-linalg. Does the compiler provide any more information?

[sstadick]

The compiler just complains that it couldn't find blas here: /usr/local/opt/openblas/lib. That path is coming rom openblas-src.

It looks like openblas-src is expecting homebrew to symlink into a specific directory: https://github.com/blas-lapack-rs/openblas-src/blob/addd8acc6a20f9e4493814eb57f7a03d7eb0ab24/openblas-src/build.rs#L44

That comment actually notes that on apple silicon it doesn't symlink, but then still uses the path as it it was symlinked.

I'm not confident enough in my build.rs / cfg! knowledge to try to fix it there.

[jturner314]

Okay, so for the openblas-system backend, openblas-src should be updated to search in another directory, or at least the docs should be updated to inform users that they may need to set LDFLAGS and CPPFLAGS.

What about the openblas-static backend, which builds OpenBLAS from source instead of using the system OpenBLAS? @M1KE-TAYL0R's initial report of this issue indicates that OpenBLAS fails to compile. Does the compiler provide any additional information other than the 101 exit status?

[sstadick]

I haven't tried openblas-static and don't have direct access to an M1 (I was debugging on a colleagues machine yesterday to get a project working for them).

[jturner314]

@sstadick Oh, okay, thanks for your help. I just assumed that you had easy access to an M1 machine.

To any M1 users who want help building OpenBLAS from source instead of using pre-built binaries: please provide additional information about the error.

[jianshu93]

Hello all,

static build (feature openblas-static) on MacOS just does not work no matter it is M1 or Intel. It seems there are some problems with compiling. I will add error soon.

Jianshu

[jianshu93]

apple_M1_static_openblas_error.txt

FYI. M1 error. I can successfully build with openblas-static on intel cpus on macOS.

Thanks,

Jianshu

[lucasjinreal]

I got same error, anyone got a solution?

the openblas installed from homebrew, the dir can not be sensed by openblas-rs

[lucasjinreal]

@sstadick I tried your solution, still link error

[mike-kfed]

I have openblas 0.3.21 installed via homebrew and using this ndarray-linalg = { version = "0.16", features = ["openblas-system"] } in my Cargo.toml works fine on Apple Silicon. However, feature openblas-static does not link on this system by default, says missing symbols (sorry cannot reproduce the error messages after I used my fix). By setting this specific environment variable to the current macOS release version MACOSX_DEPLOYMENT_TARGET=13.2 cargo build makes static compile work too. Maybe that is helpful to others.

error output as already reported by @jianshu93 was fixed in openblas 0.3.20 so that error ld: file not found: loader_path should not happen nowadays.

[jianshu93]

Hello @mike-kfed With openblas 0.3.21 home-brew, ndarray-linalg v0.15，do as what you suggested using the environment variable, I still have the following:

error: linking with cc failed: exit status: 1 | = note: LC_ALL="C" PATH="/Users/jianshuzhao/.rustup/toolchains/nightly-2023-01-07-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/bin:/Users/jianshuzhao/miniconda3/condabin:/Users/jianshuzhao/Github/bowtie2-2.5.0-macos-arm64:/opt/homebrew/bin:/Users/jianshuzhao/bin:/opt/homebrew/opt/bzip2/bin:/Users/jianshuzhao/Github/mummer-4.0.0beta5/bin:/Users/jianshuzhao/go/bin:/Users/jianshuzhao/Github/hmmer-h3-arm/bin:/opt/homebrew/opt/llvm/bin:/opt/homebrew/opt/openjdk/bin:/opt/homebrew/Cellar/coreutils/9.1/bin:/opt/homebrew/opt/ruby/bin:/opt/homebrew/bin:/opt/homebrew/sbin:/Users/jianshuzhao/.cargo/bin:/usr/local/bin:/System/Cryptexes/App/usr/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/TeX/texbin:/usr/local/go/bin:/opt/X11/bin" VSLANG="1033" ZERO_AR_DATE="1" "cc" "-arch" "arm64" "/var/folders/mr/8qybn91j5fgdbrcdmq7ckftw0000gn/T/rustciaqQOO/symbols.o" "/Users/jianshuzhao/gsearch_new/annembed/target/release/deps/embed-64a836952535cbbc.embed.10cfc42d-cgu.12.rcgu.o" "-L" "/Users/jianshuzhao/gsearch_new/annembed/target/release/deps" "-L" "/Users/jianshuzhao/gsearch_new/annembed/target/release/build/openblas-src-bbe59f222908e0d0/out/opt/OpenBLAS/lib" "-L" "/Users/jianshuzhao/.rustup/toolchains/nightly-2023-01-07-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "/var/folders/mr/8qybn91j5fgdbrcdmq7ckftw0000gn/T/rustciaqQOO/libopenblas_src-cdc29c56b7e5b1d2.rlib" "/Users/jianshuzhao/.rustup/toolchains/nightly-2023-01-07-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib/libcompiler_builtins-5aa4ec02992215b2.rlib" "-liconv" "-lSystem" "-lc" "-lm" "-L" "/Users/jianshuzhao/.rustup/toolchains/nightly-2023-01-07-aarch64-apple-darwin/lib/rustlib/aarch64-apple-darwin/lib" "-o" "/Users/jianshuzhao/gsearch_new/annembed/target/release/deps/embed-64a836952535cbbc" "-Wl,-dead_strip" "-nodefaultlibs" = note: Undefined symbols for architecture arm64: "___emutls_get_address", referenced from: _sgemv_thread_n in libopenblas_src-cdc29c56b7e5b1d2.rlib(sgemv_thread_n.o) "__gfortran_concat_string", referenced from: sormbr in libopenblas_src-cdc29c56b7e5b1d2.rlib(sormbr.o) sormlq in libopenblas_src-cdc29c56b7e5b1d2.rlib(sormlq.o) sormqr in libopenblas_src-cdc29c56b7e5b1d2.rlib(sormqr.o) ld: symbol(s) not found for architecture arm64 collect2: error: ld returned 1 exit status

It seems there are something not fixed for arm64 on M series.

Thanks,

Jianshu

[mike-kfed]

I cannot reproduce that, but you are using an older compiler than me, even though you use nightly - rustc 1.68.0 is from beginning of march, and compiling works on both my M1 laptop aswell as on my M1 based desktop. Maybe try cargo clean and using recent rustc?

[jianshu93]

I still have the same error with rust 1.70 (3.20.2023), I am using v0.15, could this be the reason?

Thanks,

Jianshu

[mike-kfed]

I tried with ndarray-linalg 0.15 worked too. Is your project opensource? I don't think we have the same conditions somehow. the only thing I can offer is to look at your code and try to see if that fails on my machines too.

[jianshu93]

Hello @mike-kfed ,

It is my collaborators' work, see here: https://github.com/jean-pierreBoth/annembed

You have to remove the features in this line in Cargo to make it run for MacOS M1: hnsw_rs = {version = "0.1.18", features=["simdeez_f"]}

three features are available when running cargo build, openblas-static is there. ndarray blas feature can be changed also to openblas, we just use the default

I am using otool -L ./binary to check whether it is static link or dynamic link.

Thanks, Jianshu

[mike-kfed]

the hnsw_rs crate should add a conditional compile flag to not try to use x86 SIMD on non-x86 platforms, even when the feature flag is on ;) Anyways, here is a fix for the build problem. It seems like the Accelerate framework link instructions don't make it all the way up to annembed. patching build.rs like so:

--- a/build.rs
+++ b/build.rs
@@ -1,6 +1,8 @@
 #[cfg(feature = "openblas-system")]
-fn main() {
-}
+fn main() {}

 #[cfg(not(any(feature = "openblas-system", feature = "netlib-system")))]
-fn main() {}
\ No newline at end of file
+fn main() {
+    #[cfg(target_os = "macos")]
+    println!("cargo:rustc-link-lib=framework=Accelerate");
+}

should do it, at least it does on my machine. Please note that this is a hack, somewhere in your dependency tree that linker info should be emitted but isn't - the correct fix would be identifying the crate that really needs it and fix its build.rs.

[jianshu93]

Hello @mike-kfed ，

The hack is to use accelerate but if we do want to use openblas static as backend for ndarray and ndarray-linalg，what would be the solution here for macos，we have to identify the crate that rely on dynamic link right？

Thanks，

Jianshu

[mike-kfed]

Hey @jianshu93 , Yes correct, you'd have to find the crate that actually needs Accelerate linked and make it emit cargo:rustc-link-lib there - this would be fixing the issue at the core instead of my band-aid hack. You can also improve on my hack though and only emit the linker info when macOS is the target and openblas-static feature is active. rustc conditional compile docs will be helpful for that.

[jianshu93]

can you please share the embed binary after compiling annembed with openblas-static feature？

Thanks， jianshu

[mike-kfed]

@jianshu93 feel free to chat with me on matrix.to (see my github profile for info) - I don't think we should hijack this thread for general programming support.

[blagowtf]

I encountered this today on 0.16 with latest MacOS. Seems like unusually high friction compared to usual Rust packages. Is there no way of providing the linalg methods by way of Accelerate.framework (or at least not require brew for openblas-system).

[jianshu93]

@blagowtf , I was also asking this question in another issue. ndarray supports Accelerate.framework it seems (I can use accelerate as backend instead of openblas-src). ndarray-linalg does not support yet. It seems trivial work since those 2 should be very similar in terms of add backend.

Thanks,

Jianshu

[jianshu93]

More importantly, the performance of openblas-src on MacOS is significantly slower than on linux due to the well known bad performance of OpenMP on MacOS (e.g. parallel for or others). Adding support for accelerate framework seems to be very useful.

Thanks,

Jianshu

[jianshu93]

The src support of accelerate framework is here: https://github.com/blas-lapack-rs/accelerate-src

Jianshu