chainweb-node icon indicating copy to clipboard operation
chainweb-node copied to clipboard

Published 20 hours ago • verified publisher icon kadena-io

Illegal instruction on Raspberry Pi aarch64

Open nonfatalexec opened this issue 3 years ago • 5 comments

I compiled the chainweb-node binary using https://github.com/kadena-io/chainweb-node/archive/refs/tags/2.12.zip on a Docker running balenalib/raspberrypi3-64-node, then when I tried to run chainweb-node on the Raspberry Pi 3 with aarch64 (matches Docker container architecture), I get an "illegal instruction" error.

This is the Docker container: docker run -it --platform linux/arm64/v8 --name raspbian-bullseye -v /usr/bin/qemu-arm-static:/usr/bin/qemu-arm-static balenalib/raspberrypi3-64-node bash

Information about Raspberry Pi 3:

$ file .cabal/bin/c*
.cabal/bin/chainweb-node: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=def0d4a884c9425827902da1bf07ad3513d5dd4b, with debug_info, not stripped
.cabal/bin/cwtool:        ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, for GNU/Linux 3.7.0, BuildID[sha1]=d5d1c0ce682fd9d80cf3b5f1fce2d02b5e4bbf59, with debug_info, not stripped
$ file /bin/bash
/bin/bash: ELF 64-bit LSB pie executable, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/ld-linux-aarch64.so.1, BuildID[sha1]=641209ff5307ca8eb85bd6368cb5f7f2e694897e, for GNU/Linux 3.7.0, stripped
$ uname -a
Linux raspberrypi 5.10.63-v8+ #1459 SMP PREEMPT Wed Oct 6 16:42:49 BST 2021 aarch64 GNU/Linux

config.yaml:

databaseDirectory: /home/crypto/chainweb-db/
chainweb:
  mining:
    coordination:
      miners:
      - account: k:xxxxxobfuscatedxxxxx
        predicate: keys-all
        public-keys:
        - xxxxxobfuscatedxxxxx
  p2p:
    peer:
      hostaddress:
        hostname: xxx.xxx.xxx.xxx
        port: 1789

Output from running chainweb-node on Raspberry Pi 3 (IP obfuscated):

$ ./.cabal/bin/chainweb-node --config-file config.yaml
2022-02-10T16:30:51.513Z [Info] [chainwebVersion=mainnet01] start ServiceDate
2022-02-10T16:30:51.513Z [Warn] [chainwebVersion=mainnet01] This version of chainweb node will stop to work at 2022-02-24 00:00:00 UTC. Please upgrade to a new version before that date.
2022-02-10T16:30:51.561Z [Info] [chainwebVersion=mainnet01] opened rocksdb in directory "/home/crypto/chainweb-db/0/rocksDb"
2022-02-10T16:30:53.386Z [Info] [chainwebVersion=mainnet01] got remote info from us-e2.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":57580}}
2022-02-10T16:30:53.404Z [Info] [chainwebVersion=mainnet01] got remote info from us-e1.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":49308}}
2022-02-10T16:30:53.409Z [Info] [chainwebVersion=mainnet01] got remote info from us-e3.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":36802}}
2022-02-10T16:30:53.418Z [Info] [chainwebVersion=mainnet01] got remote info from us-w1.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":33118}}
2022-02-10T16:30:53.431Z [Info] [chainwebVersion=mainnet01] got remote info from us-w3.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":55750}}
2022-02-10T16:30:53.444Z [Info] [chainwebVersion=mainnet01] got remote info from us-w2.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":36706}}
2022-02-10T16:30:53.533Z [Info] [chainwebVersion=mainnet01] got remote info from fr2.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":51292}}
2022-02-10T16:30:53.538Z [Info] [chainwebVersion=mainnet01] got remote info from fr3.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":52346}}
2022-02-10T16:30:53.550Z [Info] [chainwebVersion=mainnet01] got remote info from fr1.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":49502}}
2022-02-10T16:30:53.574Z [Info] [chainwebVersion=mainnet01] got remote info from jp1.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":58668}}
2022-02-10T16:30:53.594Z [Info] [chainwebVersion=mainnet01] got remote info from jp3.chainweb.com:443: {"version":"2.12.1","timestamp":1644510653,"hostaddress":{"hostname":"xxx.xxx.xxx.xxx","port":60382}}
2022-02-10T16:30:57.310Z [Warn] [chainwebVersion=mainnet01] failed to get remote info from jp2.chainweb.com:443: NodeInfoConnectionFailure (HostAddress {_hostAddressHost = jp2.chainweb.com, _hostAddressPort = 443}) (HttpExceptionRequest Request {
  host                 = "jp2.chainweb.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("X-Chainweb-Node-Version","2.12")]
  path                 = "/chainweb/0.0/mainnet01/cut"
  queryString          = ""
  method               = "HEAD"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutMicro 5000000
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}
 ConnectionTimeout)
2022-02-10T16:31:22.996Z [Info] [chainwebVersion=mainnet01] Local Peer Info: {"id":"jauykEkuqI9aJMBEB4A2BJB6gk77PThOYB4bUcHirYE","address":{"hostname":"xxx.xxx.xxx.xxx","port":1789}}
2022-02-10T16:31:23.260Z [Warn] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] HTTP server error: HandshakeFailed Error_EOF
2022-02-10T16:31:23.267Z [Warn] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] HTTP server error: HandshakeFailed Error_EOF
2022-02-10T16:31:23.387Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from us-e2.chainweb.com:443
2022-02-10T16:31:23.388Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.432Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from us-e1.chainweb.com:443
2022-02-10T16:31:23.432Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.468Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from us-w3.chainweb.com:443
2022-02-10T16:31:23.468Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.491Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from us-w1.chainweb.com:443
2022-02-10T16:31:23.491Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.561Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from us-w2.chainweb.com:443
2022-02-10T16:31:23.561Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.836Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from fr1.chainweb.com:443
2022-02-10T16:31:23.836Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.844Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from fr3.chainweb.com:443
2022-02-10T16:31:23.845Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.849Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from fr2.chainweb.com:443
2022-02-10T16:31:23.849Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:23.988Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from us-e3.chainweb.com:443
2022-02-10T16:31:23.988Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:24.020Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from jp3.chainweb.com:443
2022-02-10T16:31:24.020Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:24.926Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] reachable from jp1.chainweb.com:443
2022-02-10T16:31:24.926Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] NoContent
2022-02-10T16:31:28.018Z [Warn] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] failed to be reachabled from jp2.chainweb.com:443: ConnectionError (HttpExceptionRequest Request {
  host                 = "jp2.chainweb.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Content-Type","application/json;charset=utf-8"),("X-Chainweb-Node-Version","2.12")]
  path                 = "/chainweb/0.0/mainnet01/cut/peer"
  queryString          = ""
  method               = "PUT"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutMicro 5000000
  requestVersion       = HTTP/1.1
  proxySecureMode      = ProxySecureWithConnect
}
 ConnectionTimeout)
2022-02-10T16:31:28.019Z [Info] [chainwebVersion=mainnet01|peerId=jauykE|port=1789|host=xxx.xxx.xxx.xxx] 11 out of 12 peers are reachable
Illegal instruction

nonfatalexec avatar Feb 10 '22 17:02 nonfatalexec

some of the cryptographic libraries are probably compiled with flags that enable instruction sets that aren't available on your hardware. In particular, you may check the build options for the cryptonite and the hashes packages.

larskuhtz avatar Jun 13 '22 17:06 larskuhtz

Same problem here. I built from the sources on a Raspberry Pi 400 (Bullseye/Rasbian arm64).

dleidert avatar Jul 16 '22 23:07 dleidert

This is what gdb says:

0x0000007ff7ae491c in crc32c_arm64(unsigned int, unsigned char const*, unsigned int) () from /lib/librocksdb.so.6.11

dleidert avatar Jul 16 '22 23:07 dleidert

The issue in librocksdb has been tracked as facebook/rocksdb#7363 and a fix was pulled into the upstream project via facebook/rocksdb#7233. After applying this patch to the librocksdb version in Raspbian (6.11.4) and recompiling it, I got rid of the illegal instruction error and chainweb-node seems to run now.

dleidert avatar Jul 17 '22 19:07 dleidert

The problem has been resolved in Debian/Raspbian Bullseye (https://tracker.debian.org/news/1359515/accepted-rocksdb-6114-3deb11u1-source-into-proposed-updates-stable-new-proposed-updates/).

dleidert avatar Sep 08 '22 13:09 dleidert