use buildx, Dockerfile ADD operation Error: /tmp/.../target: 'no such file or directory'

[HAoDestiny]

Description

Info

• docker-maven-plugin version : 0.40.0
• Maven version (mvn -v) : 3.8.1

Apache Maven 3.8.1 (05c21c65bdfed0f71a2f2ada8b84da59348c4c5d)
Maven home: /Applications/IntelliJ IDEA.app/Contents/plugins/maven/lib/maven3
Java version: 17.0.3.1, vendor: Oracle Corporation, runtime: /Library/Java/JavaVirtualMachines/jdk-17.0.3.1.jdk/Contents/Home
Default locale: zh_AL_#Hans, platform encoding: UTF-8
OS name: "mac os x", version: "12.3", arch: "aarch64", family: "mac"

• Docker version : Docker Desktop 4.8.2 (79419)
• Docker info

Client:
 Context:    default
 Debug Mode: false
 Plugins:
  buildx: Docker Buildx (Docker Inc., v0.8.2)
  compose: Docker Compose (Docker Inc., v2.5.1)
  sbom: View the packaged-based Software Bill Of Materials (SBOM) for an image (Anchore Inc., 0.6.0)
  scan: Docker Scan (Docker Inc., v0.17.0)

• If it's a bug, how to reproduce : Build directly using the plugin docker:build on IDEA
• If it's a feature request, what is your use case :
• Sample project : [GitHub Clone URL]

plugin buildx config xml

...
<build>
    <buildx>
        <platforms>
            <platform>linux/amd64</platform>
        </platforms>
    </buildx>
    <dockerFile>${project.basedir}/Dockerfile</dockerFile>
</build>
...

error message:

...
[INFO] DOCKER> ------
[INFO] DOCKER> WARNING: local cache import at /Users/xxx/IdeaProjects/.../target/docker/.../cache not found due to err: could not read /Users/xxx/IdeaProjects/.../target/docker/.../cache/index.json: open /Users/xxx/IdeaProjects/..../target/docker/..../cache/index.json: no such file or directory
[INFO] DOCKER> Dockerfile:8
[INFO] DOCKER> --------------------
[INFO] DOCKER>    6 |     WORKDIR /deploy/app
[INFO] DOCKER>    7 |     # add jar
[INFO] DOCKER>    8 | >>> ADD ./target/app.jar ./app.jar
[INFO] DOCKER>    9 |     
[INFO] DOCKER> --------------------
[INFO] DOCKER> error: failed to solve: failed to compute cache key: failed to calculate checksum of ref lc9yh3lib9snvkpx651007miq::nvumb2hjfwky8awzc6ewoa3nj: failed to walk /tmp/buildkit-mount981401484/target: lstat /tmp/buildkit-mount981401484/target: no such file or directory
[INFO] DOCKER> docker --config /Users/xxx/IdeaProjects/.../target/docker/.../docker buildx rm dmp_.....
[ERROR] DOCKER> Error status (1) when building
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time:  12.449 s
[INFO] Finished at: 2022-05-31T19:40:59+08:00
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal io.fabric8:docker-maven-plugin:0.40.0:build (default-cli) on project ...: Error status (1) when building -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException

I think there may be a problem with the cache configuration under buildx. Have you also encountered this kind of problem? I have been tangled for a long time. can you guys help me thanks 😂😂😂

[saravanaa14]

I have the same problem. My Dockerfile looks like `FROM flyway/flyway:8.4.2

RUN pwd

ADD src/* /flyway/sql/ `

My pom config looks like this

<plugin> <groupId>io.fabric8</groupId> <artifactId>docker-maven-plugin</artifactId> <version>0.40.0</version> <configuration> <verbose>true</verbose> <images> <image> <name>${docker.repository.base}/ABC:${project.version}</name> <build> <buildx> <platforms> <platform>linux/arm64</platform> <platform>linux/amd64</platform> </platforms> </buildx> <contextDir>${project.basedir}</contextDir> </build> </image> </images> </configuration> <executions> <execution> <id>docker-build</id> <goals> <goal>build</goal> <goal>push</goal> </goals> <phase>package</phase> </execution> </executions> </plugin>

RUN pwd returns flyway/ which shows that the process is inside the container. Because of this, buildx process doesn't recognize the path src/ which is on my local Mac.

[jleonar]

Similar issue. error: failed to solve: failed to read dockerfile: open /tmp/buildkit-mount2878279091/Dockerfile: no such file or directory

But this only happens on the docker:push. I can actually build the image.

[rohanKanojia]

@jleonar : Which version are you using? 0.40.1 ?

[jleonar]

@rohanKanojia Yes, I am using 0.40.1

[rohanKanojia]

I thought this was fixed by https://github.com/fabric8io/docker-maven-plugin/pull/1564 @saravanaa14 are you facing this issue on 0.40.1 as well?

[jleonar]

Yes, I originally tried 0.40.0 and I couldn't even build since I specify the dockerfile. I updated to 0.40.1 and was able to build. However, the docker:push doesn't work.

[saravanaa14]

@jleonar @rohanKanojia My issues are resolved with the latest version.

Sample docker file - Downloads flyway image and copies files from ddl/* to container folder /flyway/sql. ` FROM flyway/flyway:8.4.2

ADD ddl/* /flyway/sql/ ` Sample plugin config that works

<plugin> <groupId>io.fabric8</groupId> <artifactId>docker-maven-plugin</artifactId> <version>0.40.1</version> <configuration> <verbose>true</verbose> <images> <image> <name>${docker.repository.base}/${docker.project}/${project.artifactId}:${docker.tag}</name> <build> <buildx> <platforms> <platform>linux/arm64</platform> <platform>linux/amd64</platform> </platforms> </buildx> <contextDir>${project.basedir}/src/main</contextDir> </build> </image> </images> </configuration> <executions> <execution> <id>docker-build</id> <goals> <goal>build</goal> <goal>push</goal> </goals> <phase>package</phase> </execution> </executions>

[viragtripathi]

Hi @saravanaa14 I have the exact same scenario using a Dockerfile and the build works great. I provided linux/amd64,linux/arm64 under buildx platform and I can see that generated DOCKER command has both but when I look up the manifest, it only shows amd64 as the os architecture. I pushed the image to DockerHub as well but only see amd64. Am I missing something here? How will I see artifacts with arm64 architecture?

[saravanaa14]

Did you push it as part of your build? Where you able to see if arm dependencies being pulled in logs during build process

[viragtripathi]

I pushed it manually with docker image push --all-tags <>. I did see arm64 dependencies being pulled then I reliazed that my base image didn't have arm64 support as I was using FROM azul/zulu-openjdk-alpine:11 so switched to FROM azul/zulu-openjdk-alpine:17 but I still don't see arm64 in the manifest.

[saravanaa14]

You have to push using plugin config . Your local manifest will have only have the necessary arch that matches with your computer’s processor. At least that’s what I found while trying to do manual pushed from my intel mac

[viragtripathi]

Oh I see. That makes sense. I will try to push using plugin config. I guess the base image doesn't really matter so I could still use azul/zulu-openjdk-alpine:11. Thanks!

[viragtripathi]

@saravanaa14 I added push goal but for some reason it fails in authorization with [INFO] DOCKER> #28 ERROR: server message: insufficient_scope: authorization failed [INFO] DOCKER> ------ [INFO] DOCKER> > exporting to image: [INFO] DOCKER> ------ [INFO] DOCKER> error: failed to solve: server message: insufficient_scope: authorization failed My docker credentials are same as before and stored in ~/.m2/settings.xml which is what gets used when I manually push the images.

[saravanaa14]

That’s probably outside the scope of this plugin. You should run docker login yourreponame -u username And check if that user has permissions to push

[viragtripathi]

There is nothing wrong with the user as I have been using the same user for pushing the images manually. docker login works as well. I started seeing this insufficient_scope: authorization failed error when I added the push goal in my pom.

I see others have encountered similar issue with buildkit and buildx

https://forums.docker.com/t/images-not-building-after-upgrade-to-docker-for-mac-4-4-2-73305/119999/5

https://github.com/docker/build-push-action/issues/223

https://github.com/docker/build-push-action/issues/603

[saravanaa14]

@jleonar @rohanKanojia I was finally able to replicate the issue with docker:push. Here is what I did as part of team city build process. I created 3 build steps.

• Step 1 does mvn docker:build which creates an image.
• Step 2 validates this image by running it
• Step 3 mvn docker:push if Step 2 was a success.

Expectation was that docker:push would push the image from Step 1 to artifactory. However, it doesn't recognize the image even though the build job is within the same context.

Workaround : I ended up configuring Step 3 as mvn clean install docker:push which ultimately creates a new image and pushes it to artifactory. This is not ideal, but I'm sure if this plugin is supposed to work with 2 disconnected maven processes. I would assume no, since buildx creates a new buildkit instance and the 2nd maven instance would have no idea about it.

[saravanaa14]

@rohanKanojia @chonton Sorry about the tagging. https://github.com/fabric8io/docker-maven-plugin/pull/1564 fixed Dockerfile issue when you have ADD command in it. However, when I try to change from Dockerfile to xml config in plugin, I encounter the same issue as reported originally in this thread.

<plugin> <groupId>io.fabric8</groupId> <artifactId>docker-maven-plugin</artifactId> <version>0.40.1</version> <configuration> <verbose>true</verbose> <images> <image> <name>${docker.repository.base}/${docker.project}/${project.artifactId}:${docker.tag}</name> <build> <from>flyway/flyway:8.4.2</from> <buildx> <platforms> <platform>linux/arm64</platform> <platform>linux/amd64</platform> </platforms> </buildx> <assemblies> <assembly> <name>src/main/ddl/*</name> <targetDir>/flyway/sql</targetDir> </assembly> </assemblies> </build> </image> </images> </configuration> <executions> <execution> <id>docker-build</id> <goals> <goal>build</goal> </goals> <phase>package</phase> </execution> </executions> </plugin>

This config fails with [INFO] DOCKER> error: failed to solve: lstat /tmp/buildkit-mount3453332988/src/main/ddl: no such file or directory . It is again looking for src/main/ddl inside buildkit rather than the VM/machine where build is running on.

[jleonar]

@saravanaa14 thank you for reproducing. That is exactly our workflow (we run integration tests against the image in step 2). Ultimately, your workaround will be building the image twice and ultimately pushing an untested image.

[saravanaa14]

I wrote a test to reproduce this issue. Please check it out. I do not know much about this framework to make the actual fix, so pardon me on that. https://github.com/fabric8io/docker-maven-plugin/compare/master...saravanaa14:master

[afranken]

I can confirm that docker:push still does not work. Same use-case, we first build the Docker image, then run an integration test suite against it, then push it.

[chonton]

@saravanaa14 - I don't understand how your reproducer is supposed to work. I suspect that the assembly is not including the files you want. To check if the assembly is correct, untar with tar -xf target/docker/dmp-buildx-xml-config/0.41.0-SNAPSHOT/tmp/docker-build.tar

I suspect you want something like the following in your pom:

                <assemblies>
                  <assembly>
                    <!-- name>src/main/ddl/*</name -->
                    <inline>
                      <fileSets>
                        <fileSet>
                          <includes>
                            <include>src/main/ddl/**</include>
                          </includes>
                        </fileSet>
                      </fileSets>
                    </inline>
                    <targetDir>/flyway/sql</targetDir>
                  </assembly>
                </assemblies>

[chonton]

@saravanaa14 - are all three steps executed on the same TeamCity agent? The push goal assumes that the cache created by the build goal is available on the local file system

[saravanaa14]

@saravanaa14 - are all three steps executed on the same TeamCity agent? The push goal assumes that the cache created by the build goal is available on the local file system

Yes, it is done on the same agent. We build the image, run some tests on it and when we execute the goal docker push, plug-in doesn’t recognize the old build cache and fails