iroha icon indicating copy to clipboard operation
iroha copied to clipboard

Published 20 hours ago • verified publisher icon hyperledger

Local `docker build` fails on M1 Mac

Open s8sato opened this issue 3 years ago • 11 comments

GIT commit hash

02283d16

Minimum working example

COMMIT=02283d16

git checkout $COMMIT

docker build -t 2690:base-$COMMIT -t iroha2:base -f Dockerfile.base .

docker build -t 2690:build-$COMMIT -t iroha2:build -f Dockerfile.build .

docker build -t 2690:final-$COMMIT .

Expected behaviour

Succeeds, regardless of the host platform (confirmed by e.g. uname -io)

Actual behaviour

Fails on M1 Mac

Operating system

x86_64 GNU/Linux (Succeeds to build)

LibC type and version

No response

Current environment

Source code build

Logs in JSON format

Log contents 
Replace this text with a JSON log,
so it doesn't grow too large and has highlighting.

Who can help?

@s8sato

s8sato avatar Sep 02 '22 07:09 s8sato 

bash 2690-inspect.sh before >before.log

bash 2690-inspect.sh fixme >fixme.log
2690-inspect.sh 
#!/bin/bash
set -eu

# configure

ISSUE=2690
TOOLCHAINS=()

case $1 in

before)
    COMMIT='70e782bb'
    STAGE_BASE='rust-base'
    STAGE_BUILD='builder'
    TOOLCHAINS+=('stable')
    TOOLCHAINS+=('nightly-2022-04-20')
    ;;

fixme)
    COMMIT='02283d16'
    TAG_BASE='iroha2:base'
    TAG_BUILD='iroha2:build'
    TOOLCHAINS+=('stable')
    TOOLCHAINS+=('nightly-2022-04-20')
    TOOLCHAINS+=('nightly-2022-04-20-x86_64-unknown-linux-musl')
    ;;

*)
    exit 1
    ;;

esac

# build

git checkout $COMMIT

case $1 in

before)
    docker build -t $ISSUE:base-$COMMIT --target $STAGE_BASE . -q
    docker build -t $ISSUE:build-$COMMIT --target $STAGE_BUILD . -q
    # docker build -t $ISSUE:final-$COMMIT . -q
    ;;

fixme)
    docker build -t $ISSUE:base-$COMMIT -t $TAG_BASE -f Dockerfile.base . -q
    docker build -t $ISSUE:build-$COMMIT -t $TAG_BUILD -f Dockerfile.build . -q
    # docker build -t $ISSUE:final-$COMMIT . -q
    ;;

*)
    exit 1
    ;;

esac

echo

# inspect

STAGES=('base' 'build')
for stage in ${STAGES[@]}; do
    IMAGE="$ISSUE:$stage-$COMMIT"
    echo "$IMAGE toolchains:"
    docker run $IMAGE rustup toolchain list
    echo
    if [ $1 = 'before' ] && [ $stage = 'build' ]; then
        TOOLCHAINS+=('1.60')
    fi
    for toolchain in ${TOOLCHAINS[@]}; do
        echo "$IMAGE $toolchain targets:"
        docker run $IMAGE rustup +$toolchain target list --installed
        echo
        echo "$IMAGE $toolchain components:"
        docker run $IMAGE rustup +$toolchain component list --installed
        echo
    done
done
before.log 
sha256:8d3356640c7fc9732b06161974db18054bdd2961010c095c8e4e1aac65cf8f6b
sha256:68c8396e79d446eaedfe828ffcfa4aca472e3749bd34976ff126aeeb1d1ff6ca

2690:base-70e782bb toolchains:
stable-x86_64-unknown-linux-gnu (default)
nightly-2022-04-20-x86_64-unknown-linux-gnu

2690:base-70e782bb stable targets:
wasm32-unknown-unknown
x86_64-unknown-linux-gnu

2690:base-70e782bb stable components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-src
rust-std-wasm32-unknown-unknown
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

2690:base-70e782bb nightly-2022-04-20 targets:
x86_64-unknown-linux-gnu

2690:base-70e782bb nightly-2022-04-20 components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

2690:build-70e782bb toolchains:
stable-x86_64-unknown-linux-gnu (default)
nightly-2022-04-20-x86_64-unknown-linux-gnu
1.60-x86_64-unknown-linux-gnu (override)

2690:build-70e782bb stable targets:
wasm32-unknown-unknown
x86_64-unknown-linux-gnu

2690:build-70e782bb stable components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-src
rust-std-wasm32-unknown-unknown
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

2690:build-70e782bb nightly-2022-04-20 targets:
x86_64-unknown-linux-gnu

2690:build-70e782bb nightly-2022-04-20 components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

2690:build-70e782bb 1.60 targets:
wasm32-unknown-unknown
x86_64-unknown-linux-gnu

2690:build-70e782bb 1.60 components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-std-wasm32-unknown-unknown
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu
fixme.log 
sha256:b4a464c7ce6d547c3f62996ca706048bbe9b7995cc63713d713b88c3bccd5dbe
sha256:159772d267c1434bcacd74d95c5005819454ba3014dc60714142cff88f3ac010

2690:base-02283d16 toolchains:
stable-x86_64-unknown-linux-gnu (default)
nightly-2022-04-20-x86_64-unknown-linux-gnu
nightly-2022-04-20-x86_64-unknown-linux-musl

2690:base-02283d16 stable targets:
x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl

2690:base-02283d16 stable components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
llvm-tools-preview-x86_64-unknown-linux-gnu
rust-src
rust-std-x86_64-unknown-linux-gnu
rust-std-x86_64-unknown-linux-musl
rustc-x86_64-unknown-linux-gnu

2690:base-02283d16 nightly-2022-04-20 targets:
wasm32-unknown-unknown
x86_64-unknown-linux-gnu

2690:base-02283d16 nightly-2022-04-20 components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-std-wasm32-unknown-unknown
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

2690:base-02283d16 nightly-2022-04-20-x86_64-unknown-linux-musl targets:
x86_64-unknown-linux-musl

2690:base-02283d16 nightly-2022-04-20-x86_64-unknown-linux-musl components:
cargo-x86_64-unknown-linux-musl
rust-src
rust-std-x86_64-unknown-linux-musl
rustc-x86_64-unknown-linux-musl
rustfmt-x86_64-unknown-linux-musl

2690:build-02283d16 toolchains:
stable-x86_64-unknown-linux-gnu (default)
nightly-2022-04-20-x86_64-unknown-linux-gnu
nightly-2022-04-20-x86_64-unknown-linux-musl

2690:build-02283d16 stable targets:
x86_64-unknown-linux-gnu
x86_64-unknown-linux-musl

2690:build-02283d16 stable components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
llvm-tools-preview-x86_64-unknown-linux-gnu
rust-src
rust-std-x86_64-unknown-linux-gnu
rust-std-x86_64-unknown-linux-musl
rustc-x86_64-unknown-linux-gnu

2690:build-02283d16 nightly-2022-04-20 targets:
wasm32-unknown-unknown
x86_64-unknown-linux-gnu

2690:build-02283d16 nightly-2022-04-20 components:
cargo-x86_64-unknown-linux-gnu
clippy-x86_64-unknown-linux-gnu
rust-docs-x86_64-unknown-linux-gnu
rust-std-wasm32-unknown-unknown
rust-std-x86_64-unknown-linux-gnu
rustc-x86_64-unknown-linux-gnu
rustfmt-x86_64-unknown-linux-gnu

2690:build-02283d16 nightly-2022-04-20-x86_64-unknown-linux-musl targets:
x86_64-unknown-linux-musl

2690:build-02283d16 nightly-2022-04-20-x86_64-unknown-linux-musl components:
cargo-x86_64-unknown-linux-musl
rust-src
rust-std-x86_64-unknown-linux-musl
rustc-x86_64-unknown-linux-musl
rustfmt-x86_64-unknown-linux-musl

s8sato avatar Sep 07 '22 18:09 s8sato

Clues, or might be misleads:

  • https://docs.docker.com/desktop/mac/apple-silicon/
  • The causative commit 162d8e86 introduced x86_64-unknown-linux-musl as a build target
  • https://docs.docker.com/engine/reference/commandline/buildx_build/#platform

s8sato avatar Sep 12 '22 09:09 s8sato

i think that the problem here is in cross platform musl compilation for ARM arch on Darwin platform. Using M1 self-hosted runner, I was able to compile base image on it by adding FROM --platform=linux/amd64 archlinux:base-devel line to Dockerfile.base. But here is the error while iroha compilation: buildx failed with: error: failed to solve: process "/bin/sh -c mold --run cargo build --profile deploy --target x86_64-unknown-linux-musl --features vendored" did not complete successfully: exit code: 101 I think that this approach requires to integrate macos-cross-toolchains into base image (it requited to re-do base image to allow to use brew packet manager). @s8sato what do you think? How much this is relevant?

BAStos525 avatar Sep 19 '22 11:09 BAStos525

May be we can try to use this as base image?

BAStos525 avatar Sep 19 '22 11:09 BAStos525

I was able to compile iroha without Docker on local M1 instance.

  1. Add ./cargo/config.toml with following lines:
[target.x86_64-unknown-linux-musl]
linker = "x86_64-linux-musl-gcc"
  1. Install the required linker: brew install FiloSottile/musl-cross/musl-cross
  2. Execute the following command to compile: TARGET_CC=x86_64-linux-musl-gcc cargo build --profile deploy --target x86_64-unknown-linux-musl --features vendored

I was unlucky to do the same thing inside Docker switching the base image to mac due brew in not installed here and it hadn't been successfully installed manually via RUN command. IDK if an another working way exists. We can try to install brew and this Mac linker inside the current base image but I guess that it will not work. Anyway, it requires either to modify or to create a new one base image for M1 compilation for this working solution.

BAStos525 avatar Sep 20 '22 19:09 BAStos525

Removing the need for the base image first would simplify this issue and reduce most of the subsequent tasks. In that case, developers and CI will respectively have their own layer cache. After that, if this issue is resolved by adjusting only M1 users' local environments, that's it for the original issue -- local docker build. As another issue, the published final image should be checked if it works in M1. We may have to enable multi-platform builds in docker/build-push-action

s8sato avatar Sep 21 '22 15:09 s8sato

What do you suggest to cache?

BAStos525 avatar Sep 21 '22 17:09 BAStos525

So. @Arjentix reported that the last iroha2-dev image works correctly on his M1 machine through docker-compose deployment.

BAStos525 avatar Sep 22 '22 10:09 BAStos525

There have been no recent commits to docker files that would resolve this issue. Do you think something has changed locally @Arjentix ?

s8sato avatar Sep 23 '22 05:09 s8sato

@s8sato , no, Vasily means, that downloaded image works on M1, it always was working. But I'm still unable to build an image

Arjentix avatar Sep 23 '22 08:09 Arjentix

As another issue, the published final image should be checked if it works in M1.

I meant after the local build is resolved, the corresponding remote build result will also have to be checked on the same modified docker files @BAStos525

What do you suggest to cache?

Build cache, which is locally enabled by default, while CI needs to devise

s8sato avatar Sep 23 '22 10:09 s8sato

I mean do you suggest to cache the whole CI image with all rustup and cargo dependencies?

BAStos525 avatar Sep 26 '22 06:09 BAStos525

do you suggest to cache the whole CI image with all rustup and cargo dependencies?

Yes, in the sense of replacing the base image @BAStos525 However, there are no clear boundaries, as it is automatically determined up to which layer the cache is valid. So our job would be to order layers of appropriate granularity in a single dockerfile, from the least frequently changed to the most frequently changed

s8sato avatar Sep 28 '22 08:09 s8sato

Using the simplest type=gha cache type it allows automatically keep the cache of the base-image. These cached docker layers are available only within the target branch (we can not use the same cache for iroha2-dev and iroha2-stable branches), but probably it's even better. Compilation of iroha itself is running every time from the scratch. So, should we try to keep a cache of some cargo/rustup dependencies as well? Which ones exactly? Also we need to consider if the default 10 Gb of cache that GH provides is enough in our case (give a chance to depot). Also, we can add some cache changes conditional to manage steps execution using actions/cache. @s8sato

BAStos525 avatar Sep 30 '22 09:09 BAStos525

@pesterev

pesterev avatar Oct 05 '22 09:10 pesterev

I can notice that for ex base-image cache works with cache type gha. It allows to save a some time amount of dockerhub job.

  1. the first dockerhub job without caching: 36 m 27 s
  2. the second dockerhub job with caching: 25 m 3 s
  3. the third dockerhub job with caching: 20 m s s
  4. the fourth dockerhub job with caching: 27 m 28 s

The default cache size limit up to 10 Gb that Github gives for the whole repository seems as suitable.

By the way, we don't cache any cargo crates that are needed for mold --run cargo build --profile deploy --target x86_64-unknown-linux-musl --features vendored Shall we have to try to cache them? For example, using actions/cache@v3 with some possibilities that it provides. Or it's not a good idea because of versions specifications? @s8sato

BAStos525 avatar Oct 06 '22 10:10 BAStos525

There's a third option for caching. We can use cargo lock to pre-build the docker image in CI. This can be done using e.g. cargo chef, leading to us having the full dependencies already in place. This skips the download and upload of the cache, and instead re-uses the docker image that's already on the runners.

appetrosyan avatar Oct 06 '22 11:10 appetrosyan

Since we enabled caching for builder ci-image, its approximate cache size is more than 2 Gb. And assume that the main reason why we did it it was to bypass the problem with iroha compilation on Apple M1 (it was not possible to compile Linux Arch ci-image on M1). It seems as it didn't help: using pre-saved ci-image cache that was created by the first CI job that was run on ubuntu-latest, the second CI job from the same branch that was run on M1 returned Error: Container operations are only supported on Linux runners. We might try it once more. If it doesn't help, I can't see a reason to keep the heavy builder in the limited to 10 Gb cache since we have rust-cache and we are going to implement cargo chef as well. So, my suggestion is to finish with cargo chef integration and probably to disable builder caching by returning this PR. Also, regarding to @s8sato words, it seems as it's impossible to compile iroha image specially for Apple (darwin) platform using the official docker/build-push-action: both available setup-buildx and setup-qemu options supports only linux/... based architectures.

BAStos525 avatar Oct 26 '22 10:10 BAStos525

UPD: Error: Container operations are only supported on Linux runners caused by:

container:
  image: 7272721/i2-ci:nightly

What's is really into this image and if it's possible do not use it to build ci-image? Actually we can use cached builder in order do not compile it on M1 and proceed with iroha compilation directly.

BAStos525 avatar Oct 26 '22 14:10 BAStos525

Here is logs of the failed CI job that run on the M1 with builder image cached. Could you please help to understand what's the problem and how it could be fixed?

BAStos525 avatar Oct 26 '22 16:10 BAStos525

It looks like a specific library fails to compile. We should try and run this outside docker, so we can get a clearer picture of what's going on.

appetrosyan avatar Oct 26 '22 22:10 appetrosyan

It looks like a specific library fails to compile. We should try and run this outside docker, so we can get a clearer picture of what's going on.

I dit it some time ago. Here is just a way where I was succeed among a bunch of approaches.

BAStos525 avatar Oct 27 '22 07:10 BAStos525

Closing this as superseded by #3451. Caching questions still seem to be relevant despite being somewhat addressed, but they weren't the focus of this issue anyway.

ilchu avatar Jul 26 '23 08:07 ilchu