Eclair crashes on Mac OS X M1 (arm64)

[rorp]

eclair-node.sh and the tests crash on Mac Book M1 with No native library found: at /fr/acinq/secp256k1/jni/native/darwin-aarch64/libsecp256k1-jni.dylib exception

$ uname -a
Darwin hostname.local 20.6.0 Darwin Kernel Version 20.6.0: Tue Apr 19 21:04:40 PDT 2022; root:xnu-7195.141.29~1/RELEASE_ARM64_T8101 arm64

$ mvn -version
Apache Maven 3.8.6 (84538c9988a25aec085021c365c560670ad80f63)
Maven home: /opt/homebrew/Cellar/maven/3.8.6/libexec
Java version: 18.0.2.1, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk/18.0.2.1/libexec/openjdk.jdk/Contents/Home
Default locale: en_US, platform encoding: UTF-8
OS name: "mac os x", version: "11.6.7", arch: "aarch64", family: "mac"

$ file /opt/homebrew/Cellar/openjdk/18.0.2.1/libexec/openjdk.jdk/Contents/Home/bin/java
/opt/homebrew/Cellar/openjdk/18.0.2.1/libexec/openjdk.jdk/Contents/Home/bin/java: Mach-O 64-bit executable arm64

$ mvn test

fr.acinq.eclair.wire.protocol.OfferTypesSpec *** ABORTED ***
  java.lang.RuntimeException: Unable to load a Suite class that was discovered in the runpath: fr.acinq.eclair.wire.protocol.OfferTypesSpec
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:81)
  at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
  at scala.collection.immutable.Vector2.map(Vector.scala:1886)
  at scala.collection.immutable.Vector2.map(Vector.scala:433)
  at org.scalatest.tools.DiscoverySuite.<init>(DiscoverySuite.scala:37)
  at org.scalatest.tools.Runner$.genDiscoSuites$1(Runner.scala:1131)
  at org.scalatest.tools.Runner$.doRunRunRunDaDoRunRun(Runner.scala:1225)
  at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24(Runner.scala:992)
  at org.scalatest.tools.Runner$.$anonfun$runOptionallyWithPassFailReporter$24$adapted(Runner.scala:970)
  at org.scalatest.tools.Runner$.withClassLoaderAndDispatchReporter(Runner.scala:1481)
  ...
  Cause: java.lang.ExceptionInInitializerError:
  at fr.acinq.secp256k1.Secp256k1.<clinit>(Secp256k1.kt)
  at fr.acinq.bitcoin.PrivateKey.publicKey(PrivateKey.kt:57)
  at fr.acinq.bitcoin.scalacompat.Crypto$PrivateKey.publicKey(Crypto.scala:33)
  at fr.acinq.eclair.wire.protocol.OfferTypesSpec.<init>(OfferTypesSpec.scala:33)
  at java.base/jdk.internal.reflect.DirectConstructorHandleAccessor.newInstance(DirectConstructorHandleAccessor.java:67)
  at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:499)
  at java.base/java.lang.reflect.ReflectAccess.newInstance(ReflectAccess.java:128)
  at java.base/jdk.internal.reflect.ReflectionFactory.newInstance(ReflectionFactory.java:341)
  at java.base/java.lang.Class.newInstance(Class.java:677)
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:66)
  ...
  Cause: java.lang.reflect.InvocationTargetException:
  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:119)
  at java.base/java.lang.reflect.Method.invoke(Method.java:577)
  at fr.acinq.secp256k1.Secp256k1JvmKt.tryLoad(Secp256k1Jvm.kt:26)
  at fr.acinq.secp256k1.Secp256k1JvmKt.getSecpk256k1(Secp256k1Jvm.kt:34)
  at fr.acinq.secp256k1.Secp256k1$Companion.<init>(Secp256k1.kt:161)
  at fr.acinq.secp256k1.Secp256k1$Companion.<clinit>(Secp256k1.kt)
  at fr.acinq.secp256k1.Secp256k1.<clinit>(Secp256k1.kt)
  at fr.acinq.bitcoin.PrivateKey.publicKey(PrivateKey.kt:57)
  at fr.acinq.bitcoin.scalacompat.Crypto$PrivateKey.publicKey(Crypto.scala:33)
  at fr.acinq.eclair.wire.protocol.OfferTypesSpec.<init>(OfferTypesSpec.scala:33)
  ...
  Cause: java.lang.IllegalStateException: No native library found: at /fr/acinq/secp256k1/jni/native/darwin-aarch64/libsecp256k1-jni.dylib
  at fr.acinq.secp256k1.jni.NativeSecp256k1JvmLoader.loadSecp256k1NativeLibrary(NativeSecp256k1JvmLoader.kt:203)
  at fr.acinq.secp256k1.jni.NativeSecp256k1JvmLoader.load(NativeSecp256k1JvmLoader.kt:34)
  at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
  at java.base/java.lang.reflect.Method.invoke(Method.java:577)
  at fr.acinq.secp256k1.Secp256k1JvmKt.tryLoad(Secp256k1Jvm.kt:26)
  at fr.acinq.secp256k1.Secp256k1JvmKt.getSecpk256k1(Secp256k1Jvm.kt:34)
  at fr.acinq.secp256k1.Secp256k1$Companion.<init>(Secp256k1.kt:161)
  at fr.acinq.secp256k1.Secp256k1$Companion.<clinit>(Secp256k1.kt)
  at fr.acinq.secp256k1.Secp256k1.<clinit>(Secp256k1.kt)
  at fr.acinq.bitcoin.PrivateKey.publicKey(PrivateKey.kt:57)
  ...

[t-bast]

I believe that's expected, our secp256k1 bindings don't support the M1 chip yet, @sstone is that correct?

[sstone]

Yes unfortunately it is correct, see https://github.com/ACINQ/secp256k1-kmp/issues/54 for additional info.

[rorp]

I found a workaround. Just download an x86-64 version of JVM and run Eclair with it.

[sstone]

Thanks! Is there a significant impact on runtime performance ? (though it probably does not matter that much since macos M1 would only be used as a development platform, not a production runtime).

[rorp]

I ran tests on my 2019 MacBook Pro 2.6 GHz i7 and 2020 MacBook Pro M1 with Oracle JDK 17.0.4.1 built for Intel OSX. It took a little less than 8 minutes to run eclair-core tests on i7 and a little more than 8 minutes on M1.

However, the postgresql test fail on M1 -- there's an opposite issue: the Intel built postgres cannot load the dynamic libraries built for M1. PR https://github.com/ACINQ/eclair/pull/2428 fixes this.

[sstone]

Intel JVM performance on M1 processors is really impressive! Would the postgres tests work on an M1 machine or would whey have to use docker even if we provided JNI bindings for secp256k1 on macos M1 ? (my understanding is that they would fail anyway without #2428)

[rorp]

otj-pg-embedded v0.13.3 includes postgres binaries for Linux, Windows, and Mac OS X, but for some reason libpq for Mac OS X is missing. That means that in order to run EmbeddedPostgres on Mac OS X one needs to install postgres, and they don't ship M1 postgres binaries.

[rorp]

A better workaround for a development environment.

Download an unnoficial build of darwin-aarch64 JAR file from https://github.com/rorp/secp256k1-kmp-jni-jvm-darwin/blob/master/secp256k1-kmp-jni-jvm-darwin-0.7.0.jar

Then install it to your local Maven repository:

mvn install:install-file \
   -Dfile=./secp256k1-kmp-jni-jvm-darwin-0.7.0.jar \
   -DgroupId=fr.acinq.secp256k1 \
   -DartifactId=secp256k1-kmp-jni-jvm-darwin \
   -Dversion=0.7.0 \
   -Dpackaging=jar

Dont' forget to retest everything on a supported platform like linux-x86_64.

[sstone]

Closed by https://github.com/ACINQ/eclair/pull/2658 (eclair, through https://github.com/ACINQ/bitcoin-lib, includes JNI bindings for secp256k1-kmp on macos M1)