fix: macOS e2e local tests fails on M3 silicon processors

[falcucci]

Some crucial dependencies were missing to make the cardano-cli and cardano-node to work as it should in a standalone environment.

after-all, this aims to bring more compatibility for a general local troubleshooting due to continuous errors the script has shown to start the nodes, as you can see as follows:

cargo build --release -p mithril-aggregator -p mithril-signer -p mithril-client-cli -p mithril-relay && cargo run -p mithril-end-to-end -- -vvvvvvvvvv --bin-directory target/release/ --work-directory=$(PWD)/artifacts --devnet-scripts-directory=./mithril-test-lab/mithril-devnet --cardano-node-version 8.7.3
    Finished `release` profile [optimized] target(s) in 0.20s
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
     Running `target/debug/mithril-end-to-end -vvvvvvvvvv --bin-directory target/release/ --work-directory=/Users/falcucci/txpipe/mithril/artifacts --devnet-scripts-directory=./mithril-test-lab/mithril-devnet --cardano-node-version 8.7.3`
Jun 18 23:05:32.395 INFO Bootstrapping the Devnet, script: /Users/falcucci/txpipe/mithril/mithril-test-lab/mithril-devnet/devnet-mkfiles.sh
tar: ./bin/cardano-cli: Not found in archive
tar: ./bin/cardano-node: Not found in archive
tar: Error exit delayed from previous errors.
dyld[38816]: Library not loaded: @executable_path/libsodium.23.dylib
  Referenced from: <no uuid> /Users/falcucci/txpipe/mithril/artifacts/devnet/cardano-cli
  Reason: tried: '/Users/falcucci/txpipe/mithril/artifacts/devnet/libsodium.23.dylib' (no such file), '/usr/local/lib/libsodium.23.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)
/Users/falcucci/txpipe/mithril/mithril-test-lab/mithril-devnet/mkfiles/mkfiles-cardano.sh: line 81: 38816 Abort trap: 6           $CARDANO_CLI byron genesis genesis --protocol-magic ${NETWORK_MAGIC} --start-time "${START_TIME}" --k ${SECURITY_PARAM} --n-poor-addresses 0 --n-delegate-addresses ${NUM_SPO_NODES} --total-balance ${INIT_SUPPLY} --delegate-share 1 --avvm-entry-count 0 --avvm-entry-balance 0 --protocol-parameters-file "${ARTIFACTS_DIR_TEMP}/byron.genesis.spec.json" --genesis-output-dir "${ARTIFACTS_DIR_TEMP}/byron-gen-command"
Jun 18 23:05:35.776 INFO Starting the Devnet, script: /Users/falcucci/txpipe/mithril/artifacts/devnet/start-cardano.sh
Error: Failed to start the devnet

Caused by:
    No such file or directory (os error 2)

[dlachaume]

Hi @falcucci!

Did you try running the E2E without specifying the version option for the cardano-node? The version 8.7.3 is a bit old now.

On my local environment (MacOS M2), I'm running the E2E with this command without any issue: cargo run -p mithril-end-to-end -- -vvv --bin-directory target/release/ --devnet-scripts-directory=./mithril-test-lab/mithril-devnet --work-directory=/tmp/mithril_end_to_end

[falcucci]

@dlachaume hey, it fails specifying any node version and even without it.

the cardano-node version 8.7.3 is part of the e2e tests scenarios that we have ran currently in the CI steps as the same as

• 8.9.0
• 8.11.0
• 8.1.2

So it should also be replicable.

Anyway this isn't a node-version issue specifically but the setup of the libraries itself.

cargo run -p mithril-end-to-end -- -vvv --bin-directory target/release/ --devnet-scripts-directory=./mithril-test-lab/mithril-devnet --work-directory=/tmp/mithril_end_to_end
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.34s
     Running `target/debug/mithril-end-to-end -vvv --bin-directory target/release/ --devnet-scripts-directory=./mithril-test-lab/mithril-devnet --work-directory=/tmp/mithril_end_to_end`
Jun 19 10:31:56.835 INFO Bootstrapping the Devnet, script: /Users/falcucci/txpipe/mithril/mithril-test-lab/mithril-devnet/devnet-mkfiles.sh
dyld[51555]: Library not loaded: @executable_path/libsodium.23.dylib
  Referenced from: <no uuid> /private/tmp/mithril_end_to_end/devnet/cardano-cli
  Reason: tried: '/private/tmp/mithril_end_to_end/devnet/libsodium.23.dylib' (no such file), '/usr/local/lib/libsodium.23.dylib' (mach-o file, but is an incompatible architecture (have 'arm64', need 'x86_64')), '/usr/lib/libsodium.23.dylib' (no such file, not in dyld cache)
/Users/falcucci/txpipe/mithril/mithril-test-lab/mithril-devnet/mkfiles/mkfiles-cardano.sh: line 81: 51555 Abort trap: 6           $CARDANO_CLI byron genesis genesis --protocol-magic ${NETWORK_MAGIC} --start-time "${START_TIME}" --k ${SECURITY_PARAM} --n-poor-addresses 0 --n-delegate-addresses ${NUM_SPO_NODES} --total-balance ${INIT_SUPPLY} --delegate-share 1 --avvm-entry-count 0 --avvm-entry-balance 0 --protocol-parameters-file "${ARTIFACTS_DIR_TEMP}/byron.genesis.spec.json" --genesis-output-dir "${ARTIFACTS_DIR_TEMP}/byron-gen-command"
Jun 19 10:31:59.822 INFO Starting the Devnet, script: /private/tmp/mithril_end_to_end/devnet/start-cardano.sh
Error: Failed to start the devnet

Caused by:
    No such file or directory (os error 2)

[Alenar]

Hi @falcucci !

There's one problem: when a CI fail for investigation we do an archive that include the artifact directory but exclude the binaries, but with your PR we would need to also exclude all the files that you added to the gitignore (cf. ci.yml:333 "Upload E2E Tests Artifacts" step). This is not sustainable, especially since we don't know, and don't want to know, future changes to the Cardano bin archive.

We should do as follow:

1. decompress the entire archive in a new, gitignored, bin subdirectory.
2. copy this bin subdirectory, not just its content, to the target 'artifact' directory (and leave it as is afterward, since it is reused in 'skip download' mode which allow a user to provide their own binaries).
3. adapt the scripts in mithril-test-lab/mithril-devnet/mkfiles to execute the cardano-node and cardano-cli from ./bin/ instead of ./.
4. cleanup mithril-test-lab/mithril-devnet/.gitignore and in ci.yml:333 "Upload E2E Tests Artifacts" step, since we only need to exclude one directory instead of the list of content of the Cardano-bin archive

This is quite more works, are you up for that do you want us to handle it ?

[falcucci]

@Alenar that makes total sense, moving forward to an isolated /bin is a better approach.

I can handle it when I have some free time soon 🙂