arm64 docker image tag overwrites amd64 tag

[CharlieC3]

Describe the bug On the previous release, I noticed the Docker image digest for the 2.05.0.2.0 release changed shortly after it was first created. The first image digest was for an image built for the amd64 arch, and the new one which overwrote the docker tag was for the arm64 arch.

After investigating, it seems the docker-platform.yml workflow is overwriting the amd64 Docker image pushed up in the ci workflow, with the arm64 it builds. The intention is to use the same image tag for both image archs. but this doesn't appear to be happening.

I was able to fix this because luckily I had previously pulled the amd64 image digest down from the docker hub before it was overwritten, so I was able to push it back up. But if this issue isn't corrected before the next time any repo tag is created via running the ci workflow, this is going to happen again.

Steps To Reproduce Run the ci workflow with a tag passed in as a parameter. Observe the resulting docker arch available for the tag created in docker hub.

Expected behavior A docker tag should be created which should reference two different architecture builds. Running docker pull should automatically pull down the appropriate image for the host machine. See here for an example docker tag which allows for multiple archs.

[wileyj]

subscribed. i might be able to do some testing around this soon based on our convo today

[wileyj]

Confirmed our suspicions in a test repo, i'll use that repo to try some ideas on workarounds vs building all images in the same action.

[dngrhm]

Hey @wileyj. How is this going? I have a branch ready to add an arm64/v8 stretch build to the CI workflow. I was creating a PR and found this issue. Is updating the CI build to use buildx and adding the arm64 build the right path forward? Did you find a way to work out the additional Docker builds workflow?

Alternative idea: Add amd64 to the additional Docker workflow. That way the CI build isn't building all and the amd64 build is replaced by both builds. Downfall, it builds amd64 twice.

[wileyj]

Hey @wileyj. How is this going? I have a branch ready to add an arm64/v8 stretch build to the CI workflow. I was creating a PR and found this issue. Is updating the CI build to use buildx and adding the arm64 build the right path forward? Did you find a way to work out the additional Docker builds workflow?

Alternative idea: Add amd64 to the additional Docker workflow. That way the CI build isn't building all and the amd64 build is replaced by both builds. Downfall, it builds amd64 twice.

I'd be interested in seeing the branch you have - but the problem here was never building an arm64 version, it was the amount of time it took to build the source via qemu on an arm64 with docker itself (estimates using github actions were around 60-90 minutes per build).

I've been working on a test repo myself of updating most of the github action workflows, but it's not ready for a PR - still a lot of work to get it just right.

the method i'm taking is to build the binaries for all platforms/arch we want to target (arm64-musl is a little tricky, but doable) - then building all the docker images at once, for all arch using the different binaries built from source. doing it this way seems to be the better option for speed/DRY, but there are some caveats i'm working around (we still want to have a non-tagged docker image based on the debian image for example), so a bit more work is required to get the logic ironed out.

[ghost]

You can see here that building both images in the additional Docker workflow as @dngrhm suggests took a total of 2h 26m 2s: https://github.com/stacks-network/stacks-blockchain/runs/4939061601?check_suite_focus=true

This was when @zone117x ran the build from this commit: https://github.com/stacks-network/stacks-blockchain/pull/2959/commits/9c84c17c190186b4e6bd37905fcbe2f399f4d1c4

In the end it did push both images to docker hub:

platforms was hardcoded to: platforms: linux/amd64,linux/arm64

Given this approach does seem to work, can this be a stop gap solution @wileyj ?

[wileyj]

unfortunately no, since as you mentioned: 2h 26m 2s. Take a look at the PR linked in this issue, a different approach is being proposed for merge.

On Thu, Jul 21, 2022 at 10:00 PM mutharasus @.***> wrote:

You can see here that building both images in the additional Docker workflow as @dngrhm https://github.com/dngrhm suggests took a total of 2h 26m 2s:

https://github.com/stacks-network/stacks-blockchain/runs/4939061601?check_suite_focus=true

This was when @zone117x https://github.com/zone117x ran the build from this commit: 9c84c17 https://github.com/stacks-network/stacks-blockchain/commit/9c84c17c190186b4e6bd37905fcbe2f399f4d1c4

In the end it did push both images to docker hub: [image: Screen Shot 2022-07-22 at 12 55 30 AM] https://user-images.githubusercontent.com/5003742/180365697-5b95e4eb-55ee-4913-8de6-817be8f4bea0.png

platforms was hardcoded to: platforms: linux/amd64,linux/arm64

Given this approach does seem to work, can this be a stop gap solution @wileyj https://github.com/wileyj ?

— Reply to this email directly, view it on GitHub https://github.com/stacks-network/stacks-blockchain/issues/3148#issuecomment-1192183587, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAVXIHDEMBX4CIPZUG3BEX3VVITGRANCNFSM5W6DOEOQ . You are receiving this because you were mentioned.Message ID: @.***>

[muthusleuths]

@wileyj @CharlieC3 Just wanted to check in on this. Any update on where this effort is?

[wileyj]

@wileyj @CharlieC3 Just wanted to check in on this. Any update on where this effort is?

Very soon - i've been hesitant to change the build process until 2.1 is finally released (rc4 was just created a few days, fingers crossed it's the last rc).

I've started preparing to restart work on #3199 - once 2.1 is officially released, i expect we can close this issue soon afterward.

the reason that work was paused is mainly because 2.1 is a big change, and I didn't want to introduce anything that would slow down development/troubleshooting until after it's been officially released. Once it's released and we're back to incremental updates, I'm confident we can get these changes merged fairly quickly since the heavy-lifting portion of the PR is already done (all that's left is merging 2.1 code into that PR branch and confirming everything still works as expected).

[wileyj]

@wileyj @CharlieC3 Just wanted to check in on this. Any update on where this effort is?

@muthudotbtc - PR is in a good state again, waiting on approvals. it did raise another question though that i may have to address - stacks-blockchain-api is not currently building an arm64 image either, so i'm looking into what would be involved in making that happen.

[muthusleuths]

Thanks @wileyj. Yes that make sense. I would add to the list the hirosystems/explorer as well. Let me know if I can help with this in anyway.

[wileyj]

Thanks @wileyj. Yes that make sense. I would add to the list the hirosystems/explorer as well. Let me know if I can help with this in anyway.

should be easily done - the API changes are looking easy as well, had to hardcode a bunch of stuff to build it outside of Hiro's repo, but i think i can have a PR there soon. i would imagine the explorer is being built in a similar way, and if it is i can definitey PR that unless you wanted to take it on.

the changes are minimal, essentially you have to add a line like this:

platforms: linux/arm64, linux/amd64, linux/amd64/v2, linux/amd64/v3

to the step using docker/build-push-action. testing the build can be a bit tiresome though