Multi-platform buildx: unauthorized for fetching metadata for the base image

[akersch]

Description

Using the plugin to run multi-platform builds via buildx fails to get the meta-data of base images, if the base image is located at a registry which needs authentication. The registry is configured in settings.xml.

Info

• docker-maven-plugin version : 0.40.1
• Maven version (mvn -v) :

Maven home: /opt/homebrew/Cellar/maven/3.8.5/libexec
Java version: 11.0.15, vendor: Homebrew, runtime: /opt/homebrew/Cellar/openjdk@11/11.0.14.1/libexec/openjdk.jdk/Contents/Home
Default locale: en_AT, platform encoding: UTF-8
OS name: "mac os x", version: "12.4", arch: "aarch64", family: "mac"

• Docker version : 20.10.13
• If it's a bug, how to reproduce : Build a multi-platform image with a base image from a private registry.

Sample output:

[INFO] DOCKER> #4 [linux/amd64 internal] load metadata for myregistry.example.com/openjdk:8-jre-slim
[INFO] DOCKER> #4 ERROR: pulling from host myregistry.example.com failed with status code [manifests 8-jre-slim]: 401 Unauthorized
[INFO] DOCKER> 
[INFO] DOCKER> #5 [linux/arm64 internal] load metadata for myregistry.example.com/openjdk:8-jre-slim
[INFO] DOCKER> #5 ERROR: pulling from host myregistry.example.com failed with status code [manifests 8-jre-slim]: 401 Unauthorized

Using docker buildx directly on the command-line works.

[rhuss]

Sound like a valid bug. I have no insight how the buildx support works, but I suppose it uses a similar authentication mechanism like docker:build. One probably would need to look deeper into the buildx support to treat authentication the same level as "normal" builds.

[JesperTerkelsen]

I have a similar problem on docker:push using buildx, where the authentication tokens i have working in the docker config works fine, but because the buildx builder uses some kind of copied config folder, i get authentication failures when pushing.

[arauchberger]

same on my side! i was happy to see buildx support being added to the plugin - was waiting hard for it. no I'm stuck with authentication.

[arauchberger]

hi again, can it really be that since 0.40.0 (2022-05-29) nobody else ever used the build feature with a private docker registry?

don't get me wrong - i'm not complaining, i just wonder if i missed something.

i would really need help on that. thanks in advance

[rohanKanojia]

@chonton : polite ping, Do you have any idea about this?

[arauchberger]

some more details from my side: this is the command the plugin is executing:

docker \                                                                                                                                 
  --config /Users/andiy/git/at-pineit/fhirstore/product-pitdata/target/docker/index.docker.io/pineit/at.pineit.pitdata/latest/docker \
  buildx build \
  --progress=plain \
  --builder maven \
  --platform linux/arm64 \
  --tag index.docker.io/pineit/at.pineit.pitdata:latest \
  --tag index.docker.io/pineit/at.pineit.pitdata:1.7.0-SNAPSHOT \
  --tag index.docker.io/pineit/at.pineit.pitdata:1.7.0-SNAPSHOT \
  --tag index.docker.io/pineit/at.pineit.pitdata:1.7.0-SNAPSHOT-localbuild \
  --build-arg buildAppDesc="netty based microservice pitdata" \
  --build-arg buildAppName="product-pitdata" \
  --build-arg buildBranch="build-multi-arch-container-images_CU-2ewupkn" \
  --build-arg buildCommitId="a966564" \
  --build-arg buildDate="2023-04-25T07:37:04Z" \
  --build-arg buildHost="freetibet.ytec.int" \
  --build-arg buildImageTag="1.7.0-SNAPSHOT" \
  --build-arg buildOS="Mac OS X 13.3.1 (aarch64)" \
  --build-arg buildOrg="pineIT GmbH." \
  --file=/Users/andiy/git/at-pineit/fhirstore/product-pitdata/target/docker/index.docker.io/pineit/at.pineit.pitdata/latest/tmp/docker-build/Dockerfile.mini-jre11 \
  /Users/andiy/git/at-pineit/fhirstore/product-pitdata/target/docker/index.docker.io/pineit/at.pineit.pitdata/latest/tmp/docker-build \
  --load

when i try this on the command line i get exactly the same error. but when i remove the --config argument and the --builder argument i can successfully pull the private base image that is referenced in the docker file.

docker \
  buildx build \
  --progress=plain \
  --platform linux/arm64 \
  --tag index.docker.io/pineit/at.pineit.pitdata:latest \
  --tag index.docker.io/pineit/at.pineit.pitdata:1.7.0-SNAPSHOT \
  --tag index.docker.io/pineit/at.pineit.pitdata:1.7.0-SNAPSHOT \
  --tag index.docker.io/pineit/at.pineit.pitdata:1.7.0-SNAPSHOT-localbuild \
  --build-arg buildAppDesc="netty based microservice pitdata" \
  --build-arg buildAppName="product-pitdata" \
  --build-arg buildBranch="build-multi-arch-container-images_CU-2ewupkn" \
  --build-arg buildCommitId="a966564" \
  --build-arg buildDate="2023-04-25T07:37:04Z" \
  --build-arg buildHost="freetibet.ytec.int" \
  --build-arg buildImageTag="1.7.0-SNAPSHOT" \
  --build-arg buildOS="Mac OS X 13.3.1 (aarch64)" \
  --build-arg buildOrg="pineIT GmbH." \
  --file=/Users/andiy/git/at-pineit/fhirstore/product-pitdata/target/docker/index.docker.io/pineit/at.pineit.pitdata/latest/tmp/docker-build/Dockerfile.mini-jre11 \
  /Users/andiy/git/at-pineit/fhirstore/product-pitdata/target/docker/index.docker.io/pineit/at.pineit.pitdata/latest/tmp/docker-build \
  --load

i hope this can help a little bit to narrow down the problem.

[arauchberger]

the plugin is also successfully creating a config.json in --config /Users/andiy/git/at-pineit/fhirstore/product-pitdata/target/docker/index.docker.io/pineit/at.pineit.pitdata/latest/docker holding my docker credentials!

[chonton]

@arauchberger and @JesperTerkelsen: What Authentication method are you using? Do you have split push vs pull authentication? What kind of repository are you using? (ECR, gcp, artifactory, dockerhub)

[arauchberger]

hi, i'm providing docker usr/pwd via sysProp when calling maven (via GitHub-workflow). i try to pull/push images from/to dockerHub. i don't think that i have split push and pull auth. i have also tried to use a public base image to build, then i get the auth-error later when the plugin tries to push the created image.

i can see the config.json is created with the correct credentials.

[chonton]

So it's a docker bug?

[arauchberger]

i'm not sure. i tried to explain as good as i could in this comment https://github.com/fabric8io/docker-maven-plugin/issues/1583#issuecomment-1521413609. i think there must be some difference between running the docker command on the shell directly and when executed from within the maven run. if you tell me how, i can provide more debug.

[robfrank]

I'm facing a similar problem. In samples/multi-architecture when building the project :

docker --config /<long path>/docker-maven-plugin/samples/multi-architecture/target/docker/hello/multiarch/0.43-SNAPSHOT/docker buildx create --driver docker-container --name maven
[INFO] DOCKER> unknown flag: --driver
[INFO] DOCKER> See 'docker --help'.

If I remove the --confg /path all works fine. So I modified this line: https://github.com/fabric8io/docker-maven-plugin/blob/043a5698b1408e3a1a95a2e84dfed08da2ec5b24/src/main/java/io/fabric8/maven/docker/service/BuildXService.java#L66

to:

        List<String> buildX = Arrays.asList("docker",  "buildx");

And the samples/multi-architecture is working fine. Is this --config needed? To me, it looks like the bouldX service is mixing docket config and buildx config.

Final note, my versions

 docker --version
Docker version 23.0.5, build bc4487a59e
docker buildx version
github.com/docker/buildx v0.10.4 c513d34049e499c53468deac6c4267ee72948f02

[arauchberger]

hi @robfrank , thanks for this input. Building multi-arch with this patch now works as expected. For pushing the multi-arch images i had to append "--node", builderName + "0" to line 218 of BuildXService.java

https://github.com/fabric8io/docker-maven-plugin/blob/043a5698b1408e3a1a95a2e84dfed08da2ec5b24/src/main/java/io/fabric8/maven/docker/service/BuildXService.java#L218

so it now looks like this:

append(cmds, "create", "--driver", "docker-container", "--name", builderName, "--node", builderName + "0");

[arauchberger]

i'm not very familiar with opening pull-requests and i'm not sure if this now is the perfect solution, but i would be happy if somebody else, more related to this code can have a look and trigger what's needed to get a new version or snapshot publicly available.

many thanks in advance

[robfrank]

I can provide a PR with this two fixes to the project today or tomorrow

[arauchberger]

sad to see that PR https://github.com/fabric8io/docker-maven-plugin/pull/1673 didn't make it to the latest release 😢

[rohanKanojia]

@arauchberger @robfrank : Hi, I just released v0.43.2 to maven central. Could you please check if this issue is still reproducible with new version?

[arauchberger]

Hi @rohanKanojia, for my purpose 0.43.2 seems to work as expected. the --config parameter is no longer present, and therefor a local docker login is sufficient to let the plugin build multi-arch images and push them to a registry with authentication.

thanks you very much.