GoogleContainerTools
jib maven plugin build fails sometime with PATCH 404 not found
Environment:
- Jib version: 3.1.2
- Build tool: Apache Maven 3.6.3
- OS: OS name: "linux", version: "5.4.104+", arch: "amd64", family: "unix"
Description of the issue:
maven jib:build sometimes fails to upload an image layer with "PATCH https://artifactory/v2/dkr-repo/image/blobs/uploads/98fb9ada-ee1b-481b-aac4-cf67c74f850a" 404 Not Found
Expected behavior:
jib:build should always upload all layers correctly.
Steps to reproduce:
- Run "mvn --batch-mode --update-snapshots verify jib:build"
- Most times it just works, sometimes above 404 occurs.
Log output:
here an extract with "mvn -X" from a failing run:
[DEBUG] TIMED Building container configuration : 2.0 ms [DEBUG] TIMING Building a manifest list or a single manifest [DEBUG] TIMING Pushing container configuration [DEBUG] Building a single manifest [DEBUG] Skipping push; BLOB already exists on target registry : digest: sha256:238055cb317666d757f863e70fa7758c0dc66de93fcab48c57bfd72b926ff333, size: 34965462 [DEBUG] TIMED Pushing BLOB digest: sha256:238055cb317666d757f863e70fa7758c0dc66de93fcab48c57bfd72b926ff333, size: 34965462 : 21.0 ms [DEBUG] TIMED Building a manifest list or a single manifest : 47.0 ms [DEBUG] TIMING Pushing BLOB digest: sha256:99a10fea2b6685fadf842bff487525906e60cebeefd16a14ba083ba675eba369, size: 3493 [DEBUG] TIMING Checking existence of manifest [DEBUG] Checking existence of manifest for sha256:e35bfc69fe2cc912ee2d68c01d0d1f0ea315c917afc9c6554cbd87f273eaa246... [DEBUG] Skipping manifest existence check; system property set to false [DEBUG] TIMED Checking existence of manifest : 1.0 ms [DEBUG] TIMING pushBlob [DEBUG] TIMING pushBlob POST sha256:99a10fea2b6685fadf842bff487525906e60cebeefd16a14ba083ba675eba369 [DEBUG] TIMED pushBlob PATCH sha256:99a10fea2b6685fadf842bff487525906e60cebeefd16a14ba083ba675eba369 : 30.0 ms [DEBUG] TIMED pushBlob POST sha256:99a10fea2b6685fadf842bff487525906e60cebeefd16a14ba083ba675eba369 : 137.0 ms [DEBUG] TIMED pushBlob : 167.0 ms [DEBUG] TIMED Pushing BLOB digest: sha256:99a10fea2b6685fadf842bff487525906e60cebeefd16a14ba083ba675eba369, size: 3493 : 197.0 ms [DEBUG] TIMED Pushing container configuration : 255.0 ms [DEBUG] TIMED Building and pushing image : 3887.0 ms
Error: Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.1.2:build (default-cli) on project java-project: Tried to push BLOB for artifactory/dkr-repo/image with digest sha256:99a10fea2b6685fadf842bff487525906e60cebeefd16a14ba083ba675eba369 but failed because: unknown: Item dkr-repo:image/_uploads/98fb9ada-ee1b-481b-aac4-cf67c74f850a does not exist: 404 Not Found Error: PATCH https://artifactory/v2/dkr-repo/image/blobs/uploads/98fb9ada-ee1b-481b-aac4-cf67c74f850a Error: { Error: "errors" : [ { Error: "status" : 404, Error: "message" : "Item dkr-repo:image/_uploads/98fb9ada-ee1b-481b-aac4-cf67c74f850a does not exist" Error: } ] Error: } Error: -> [Help 1]
Additional Information: I'm happy to provide more debug log output if you need.
The patch location (the upload URL with the digest given by the registry)
https://artifactory/v2/dkr-repo/image/blobs/uploads/98fb9ada-ee1b-481b-aac4-cf67c74f850a
is what the registry informed Jib to use just before uploading, so it doesn't make sense that the registry says 404 (i.e., the page/URL doesn't exist). Jib simply takes and uses the given patch URL (Location: HTTP header) returned by the registry. It's like the registry saying "hey, use this URL to upload. (Jib tries to use the URL.) What? Don't use that URL, 'cause that's wrong." I highly suspect an issue in Artifactory. I suggest filing a bug there.
I'm happy to provide more debug log output if you need.
I think it is already fairly apparent for them to believe that their registry may be misbehaving. However, if we captured detailed HTTP traffic, I'm sure we should have been able to prove that conclusively (which we want to do). If possible, follow these instructions to log HTTP communications. But probably without -Djib.serialize=true, since you may not be able to reproduce this when turning off concurrency.
BTW, I also noticed that this 404 happened when uploading a container config. We have other reports where we suspect there's an issue in Artifactory when uploading a container config: #3146. One of the users opened a ticket against Artifactory, although I'm not sure if the two issues are related.
