v1.9.0 broke pushing multi-architecture images

[ChrisHines]

Describe the bug

The support for pushing multi-arch OCI images added in https://github.com/concourse/registry-image-resource/pull/321 seems to be broken in the new v1.9.0 release published yesterday.

Reproduction steps

1. Build a multi-arch image with concourse/oci-build-task
2. Push it with a put step using concourse/registry-image-resource:1.9.0 (specifying the directory created by the above step as described in https://github.com/concourse/registry-image-resource/pull/321.
3. Observe the following error:

   ERRO[0000] could not load image from path 'image/image': read /tmp/build/put/image/image: is a directory
   

4. Pin to concourse/registry-image-resource:1.8.0
5. Observe pushing the image now works.

Expected behavior

concourse/registry-image-resource:1.9.0 should work with multi-arch images just as the prior release, or there should be documentation to explain the necessary pipeline changes to get it to work.

Additional context

No response

[neodem]

I ran into this same issue. Based on the documentation and the demonstrated functionality of the resource that it expects to see the image param point to an actual tarball.. ex:

      - put: docker-hub
        params:
          image: image/image.tar

however, it seems that previous versions would accept an exploded tarball as well:

      - put: docker-hub
        params:
          image: image/image
        get_params:
          format: oci

(the above works in 1.8.0 but not in later versions).

The problem is that the oci-build-task does not include the manifest in its outputted tarball. I would say that's the original sin here.

[tgoodsell-tempus]

@neodem FYI, the exclusion of manifest.json is not a mistake, the current spec for image layouts suggests that the old manifest.json becomes embedded in blobs, while the OCI spec has specific index.json and oci-layout files at the root.

See: https://github.com/opencontainers/image-spec/blob/main/image-layout.md

Therefore, at least based on this information, registry-image should not fail on OCI built images where that manifest.json is not in the TAR/image root.

[tgoodsell-tempus]

Related upstream issue: https://github.com/google/go-containerregistry/issues/1756

My investigations so far lend me to an early conclusion that the tar reader for images is hardcoded to the "single arch image" style output.

[tgoodsell-tempus]

Also see: https://github.com/google/go-containerregistry/issues/1793

[tgoodsell-tempus]

FYI: https://github.com/google/go-containerregistry/issues/1809

[tgoodsell-tempus]

FYI, I have confirmed things are working for my sample use case on Concourse 7.9.1, hardcoding the registry image resource to 1.9.0:

See:

resource_types:
  - name: registry-image
    type: registry-image
    privileged: true
    check_every: 24h
    source:
      repository: concourse/registry-image-resource
      tag: 1.9.0


resources:
  - name: test-pipeline
    type: git
    check_every: 24h
    webhook_token: concourse
    source:
      uri: [email protected]:example/example.git
  - name: test-image
    type: registry-image
    source:
      repository: us-docker.pkg.dev/example/fly
      username: example
      password: example
jobs:
  - name: test-pipeline
    plan:
      - get: test-pipeline
        trigger: true
      - task: build-image-task
        privileged: true
        config:
          platform: linux
          image_resource:
            type: registry-image
            source:
              repository: concourse/oci-build-task
          inputs:
          - name: test-pipeline
          outputs:
          - name: image
          params:
            CONTEXT: test-pipeline/exampleDockerfile
            IMAGE_PLATFORM: linux/arm64,linux/amd64
            OUTPUT_OCI: true
          run:
            path: build
      - put: test-image
        params:
          image: image/image
          version: 0.0.1

[ChrisHines]

@tgoodsell-tempus Thanks for looking into this issue and sharing what you found. It is interesting that it works for you. Unfortunately I don't see many differences between your pipeline above and my pipeline that broke with 1.9.0. Here are the non-cosmetic differences I do see:

• I do not have privileged: true on my registry-image resource_type. I don't know why that would make a difference. It works without it using 1.8.0. (Note, I do have privileged: true on the build task, the same as you.)
• I do have inputs: detect on put step. We put that on all of our put steps because it improves performance in pipelines with a lot of resources by only streaming in the resources actually used by the put step. The error message I received suggests the image/image input was created, so this difference seems unlikely to matter.
• You have Concourse 7.9.1 and my org has 7.6.0 (probably an internal fork of that actually). I don't know if 1.9.0 of the registry image resource type would work worse with my version of Concourse for any reason.