[master] fix and refactor static packages

[crazy-max]

fixes an issue introduced by #654 while adding buildx static package. also review the bits that generates the static packages:

• with dockerd we need to detect if cross compilation is necessary. this is something we should fix upstream so we don't need this check and can full rely on --platform with buildx.
• makefile has been simplified so we don't need to duplicate goals and rely only on TARGETPLATFORM.
• Jenkinsfile is lighter and additionally generates static packages for additional platforms that were missing (linux/arm/v6, linux/arm/v7, linux/arm64)
• adds a job matrix on GHA so we can also test its behavior on GitHub Runners.
• static packages were missing for buildx against windows and darwin platforms.

Signed-off-by: CrazyMax [email protected]

[crazy-max]

moby engine cross compilation fails for linux/arm/v6. Should be fixed upstream: https://github.com/docker/docker-ce-packaging/runs/5737440583?check_suite_focus=true#step:4:2372

#43 0.132 
#43 0.138 ---> Making bundle: cross (in /build/bundles/cross)
#43 0.139 Cross building: /build/bundles/cross/linux/arm/v6
#43 0.182 Building: /build/bundles/cross/linux/arm/v6-daemon/dockerd-0.0.0-20220329083112-174e51c
#43 0.182 GOOS="linux" GOARCH="arm" GOARM="6"
#43 76.80 # github.com/docker/docker/cmd/dockerd
#43 76.80 loadinternal: cannot find runtime/cgo
#43 76.80 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
#43 76.80 gcc: error: unrecognized command-line option '-marm'; did you mean '-mabm'?

[crazy-max]

moby engine cross compilation fails for linux/arm/v6. Should be fixed upstream: https://github.com/docker/docker-ce-packaging/runs/5737440583?check_suite_focus=true#step:4:2372

#43 0.132 
#43 0.138 ---> Making bundle: cross (in /build/bundles/cross)
#43 0.139 Cross building: /build/bundles/cross/linux/arm/v6
#43 0.182 Building: /build/bundles/cross/linux/arm/v6-daemon/dockerd-0.0.0-20220329083112-174e51c
#43 0.182 GOOS="linux" GOARCH="arm" GOARM="6"
#43 76.80 # github.com/docker/docker/cmd/dockerd
#43 76.80 loadinternal: cannot find runtime/cgo
#43 76.80 /usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
#43 76.80 gcc: error: unrecognized command-line option '-marm'; did you mean '-mabm'?

fixed by https://github.com/moby/moby/pull/43474

[thaJeztah]

Opened https://github.com/crazy-max/docker-ce-packaging/pull/1 with my nits / suggestions

[crazy-max]

Looks like TARGETPLATFORM is a required parameter; wondering if we can set some sensible default for it, so that make static "works".

Yes we can evaluate the current platform by default

I also tried this PR with a specific reference, using the REF variable (which should be used for both the CLI and Engine);

make REF=v22.06.0-beta.0 TARGETPLATFORM=linux/amd64 static

Hum good point, if ref is a tag we should take that one as version for tars. I will improve that case.

[thaJeztah]

One thing we need to look into (looks to be an issue in the moby/moby repository for building containerd statically); we saw a similar warning on the CLI: https://github.com/docker/cli/pull/3494

=> => # # github.com/containerd/containerd/cmd/containerd
 => => # /usr/bin/ld: /tmp/go-link-886578672/000019.o: in function `New':
 => => # /tmp/tmp.ULyoQKWtCj/src/github.com/containerd/containerd/vendor/github.com/miekg/pkcs11/pkcs11.go:77: warning: Using 'dlopen' in statically linked applications requires at runtime the shared lib
 => => # raries from the glibc version used for linking
 => => # + bin/containerd-stress
 => => # + bin/containerd-shim

[thaJeztah]

@crazy-max I opened https://github.com/crazy-max/docker-ce-packaging/pull/2 to split the CLI, engine and containerd packages;

make REF=v22.06.0-beta.0 VERSION=v22.06.0-beta.0 TARGETPLATFORM=linux/amd64 static

static/build
├── bundles-ce-static-linux-x86_64.tar.gz
└── linux
    └── amd64
        ├── containerd-1.6.4.tgz
        ├── docker-buildx-plugin-0.8.2.tgz
        ├── docker-cli-22.06.0-beta.0.tgz
        ├── docker-compose-plugin-2.6.1.tgz
        ├── docker-engine-22.06.0-beta.0.tgz
        ├── docker-rootless-extras-22.06.0-beta.0.tgz
        └── docker-scan-plugin-0.17.0.tgz

2 directories, 8 files

ls -lh static/build/linux/amd64/
total 215208
-rw-r--r--  1 sebastiaan  staff    31M Jun 29 00:21 containerd-1.6.4.tgz
-rw-r--r--  1 sebastiaan  staff    14M Jun 29 00:21 docker-buildx-plugin-0.8.2.tgz
-rw-r--r--  1 sebastiaan  staff   8.2M Jun 29 00:21 docker-cli-22.06.0-beta.0.tgz
-rw-r--r--  1 sebastiaan  staff   8.8M Jun 29 00:21 docker-compose-plugin-2.6.1.tgz
-rw-r--r--  1 sebastiaan  staff    19M Jun 29 00:21 docker-engine-22.06.0-beta.0.tgz
-rw-r--r--  1 sebastiaan  staff    19M Jun 29 00:21 docker-rootless-extras-22.06.0-beta.0.tgz
-rw-r--r--  1 sebastiaan  staff   4.4M Jun 29 00:21 docker-scan-plugin-0.17.0.tgz

[crazy-max]

Thanks a bunch for the review @tianon!

Wouldn't it be better (and a lot simpler) to use go env GOARCH / go env GOOS here? :innocent:

If we don't get to assume access to go, maybe something like docker version --format '{{ .Server.Os }}/{{ .Server.Arch }}'? (not sure how we reliably get "current variant" but that's pretty complicated/a guess regardless, so letting the auto-detection default to v7 seems pretty sane IMO :see_no_evil:)

A bunch of the os, arch and variant detection logic is taken from xx which has bats tests to cover these cases. This project is used as cross comp helper in some of our repos. See https://github.com/tonistiigi/xx/blob/master/base/xx-info to have a better understanding of the logic. Maybe it would be better to fetch the scripts instead of just copying part of its logic. Will improve that and see if we can get rid of scripts/target-platform.

[thaJeztah]

So, looks like something's still broken; the cross-compiled versions are missing the containerd binaries. It looks like --target=cross in the moby repository does not include them in the bundles directory (need to check why)

docker buildx build \
    --build-arg CGO_ENABLED= \
    --build-arg CONTAINERD_VERSION \
    --build-arg CROSS=true \
    --build-arg DEFAULT_PRODUCT_LICENSE \
    --build-arg DOCKER_CROSSPLATFORMS=linux/arm64 \
    --build-arg PACKAGER_NAME \
    --build-arg PLATFORM \
    --build-arg PRODUCT \
    --build-arg RUNC_VERSION \
    --build-arg VERSION=22.06.0-beta.1 \
    --output ./bundles \
    --target cross .

tree bundles
bundles
└── cross
    └── linux
        └── arm64-daemon
            ├── docker-proxy -> docker-proxy-22.06.0-beta.1
            ├── docker-proxy-22.06.0-beta.1
            ├── docker-proxy-22.06.0-beta.1.md5
            ├── docker-proxy-22.06.0-beta.1.sha256
            ├── dockerd -> dockerd-22.06.0-beta.1
            ├── dockerd-22.06.0-beta.1
            ├── dockerd-22.06.0-beta.1.md5
            └── dockerd-22.06.0-beta.1.sha256

3 directories, 8 files

[thaJeztah]

I dug a bit further, and currently;

• the cross target on moby/moby only cross-compiles the docker daemon. The confusing bit was that containerd and runc are also built as part of the cross target, but not cross-compiled
• I could make containerd cross-compile by passing the right env-vars, but runc looks to need a bit more work

So, I'll have to dig further to see how feasible that is (given that we hope to build both as part of containerd packaging soon, so it would be temporary).

For platforms that we have machines for in Jenkins this won't be an issue (linux/amd64 and linux/arm64), but for other platforms that may be an issue (arm32 variants including)

[crazy-max]

@thaJeztah https://github.com/moby/moby/pull/43529 would fix this