Multi-platform build failing due to outdated skopeo in ubuntu-latest

[aaronadamsCA]

A basic multi-platform build workflow currently fails:

workflow.yml

on: [push]
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v3
      - uses: docker/setup-qemu-action@v2
        with:
          platforms: arm64
      - uses: docker/setup-buildx-action@v2
      - uses: docker/login-action@v2
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ github.token }}
      - uses: devcontainers/[email protected]
        with:
          imageName: ghcr.io/${{ github.repository }}/devcontainer
          platform: linux/amd64,linux/arm64

The error:

  Error: Dev container build failed: Command failed: docker tag vsc-repository-d8fa4a8f8a3fefdc0feb211b3b07a31c-features ghcr.io/organization/repository/devcontainer:latest (exit code: undefined)
  An error occurred building the container.
  Error: Command failed: docker tag vsc-repository-d8fa4a8f8a3fefdc0feb211b3b07a31c-features ghcr.io/organization/repository/devcontainer:latest

Single-platform builds are working fine.

I don't think this is relevant, but just in case:

.devcontainer/devcontainer.json

{
  "image": "mcr.microsoft.com/devcontainers/base:jammy",
  "features": {
    "ghcr.io/devcontainers/features/node:1": {}
  },
  "updateContentCommand": ".devcontainer/install.sh"
}

[natescherer]

Hi @aaronadamsCA, I suspect that this might be an incompatability between a new version of Docker BuildX and the older version of Skopeo available on the action runner images. Can you try adding this before the build step and see if it works?

      - name: Install updated Skopeo
        # This can be omitted once runner images have a version of Skopeo > 1.4.1
        # See https://github.com/containers/skopeo/issues/1874
        run: |
          sudo apt purge buildah golang-github-containers-common podman skopeo
          sudo apt autoremove --purge
          REPO_URL="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable"
          source /etc/os-release
          sudo sh -c "echo 'deb ${REPO_URL}/x${NAME}_${VERSION_ID}/ /' > /etc/apt/sources.list.d/devel:kubic:libcontainers:unstable.list"
          sudo wget -qnv https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable/x${NAME}_${VERSION_ID}/Release.key -O Release.key
          sudo apt-key add Release.key
          sudo apt-get update
          sudo apt-get install skopeo

[aaronadamsCA]

@natescherer I tried this just now, but the result is the same.

[eitsupi]

Could it have something to do with not setting up the push option? We can't keep multi-platform built images locally (on Docker), can we?

[baldwicc]

FWIW, I'm getting the same docker tag failure with the Azure DevOps task example

[metaskills]

I just got to this issue from a diff skopeo error.

2023-03-16T19:03:47.1593665Z Copying blob sha256:0634d0a905f9779487b6a35339b867602878bcfe541ac60331bb706c1b12540c
2023-03-16T19:03:48.6288258Z time="2023-03-16T19:03:48Z" level=fatal msg="creating an updated image manifest: preparing updated manifest, layer \"sha256:b0d6db02944d51231a28352b5c6372611214ea1a5e886d49625724204dc5c0eb\": unsupported MIME type for compression: application/vnd.in-toto+json"
2023-03-16T19:03:48.6383712Z ##[error]Error: skopeo copy failed with 1

My configs are pretty simple.

- name: Build & Push Container
  uses: devcontainers/[email protected]
  with:
    push: always
    imageName: ghcr.io/customink/some-devcontainer-service
    cacheFrom: ghcr.io/customink/some-devcontainer-service
    subFolder: .devcontainer/service
    platform: linux/amd64,linux/arm64

I can build a single platform (any of the two) but not both.

[metaskills]

@natescherer Were you hinting at this issue? https://github.com/containers/skopeo/issues/1874 I found it when I googled my error. I found that ubuntu-latest has version 1.4.1. I applied your update and my issue disappeared.

[metaskills]

Correction, the skopeo update did not work after I deleted my container package. Think it only worked because of some caching. Once I deleted my package and tried again with the devcontainers/ci workflow snippet above, things broke again and nothing helped.

[stuartleeks]

@metaskills - the config we use for the GitHub tests is here, including the skopeo install steps.

@baldwicc - the corresponding Azure DevOps test is here

[zydou]

I have also encountered this docker tag error despite having updated the skopeo package. After several attempts, I found that this issue arises only if your devcontainer.json is image based in conjunction with devcontainer-features.

For example, the following devcontainer.json will fail with the docker tag error:

{
  "image": "mcr.microsoft.com/devcontainers/base:jammy",
  "features": {
    "ghcr.io/devcontainers/features/node:1": {}
  }
}

A temporary workaround is to change your devcontainer.json to Dockerfile based. To fix the error:

1. Change your devcontainer.json to Dockerfile based.

{
  "build": {
    "dockerfile": "./Dockerfile",
    "context": "."
  },
  "features": {
    "ghcr.io/devcontainers/features/node:1": {}
  }
}

2. Create a Dockerfile file in the same dir with the devcontainer.json file

FROM mcr.microsoft.com/devcontainers/base:jammy

[stuartleeks]

Interesting. Thanks for the investigation @zydou !

Until this is fixed, there's a helpful extension that @bamurtaugh created for converting image-based dev containers to Dockerfile: https://marketplace.visualstudio.com/items?itemName=Brigit.devcontainer-image-convert

[aaronadamsCA]

I am happy to confirm that @zydou's finding is an effective workaround for me as well.

[aaronadamsCA]

Actually, it turns out @natescherer's Skopeo workaround was necessary as well.

It looks like we actually have two different issues being tracked here, then:

• My originally reported docker tag crash.
  • Can be worked around by using a Dockerfile.
• skopeo copy crash.
  • Can be worked around by upgrading Skopeo as a job step.
  • Can apparently be fixed by adding docker build --sbom=false --provenance=false args.

[stuartleeks]

The failure with image based dev containers needs more investigation to determine the underlying cause.

For the skopeo copy issue, there are a couple of things that we could do:

1. Check the skopeo version and output a message if it is outdated (and point to the docs)
2. Consider adding a skipAttestation option (this would need to be added to the CLI first)

[aaronadamsCA]

Also worth noting I tried to work around this by setting build.args to { "--sbom": "false", "--provenance": "false" }, but then the post-action skopeo copy command failed with fatal error unsupported MIME type for compression.

So it looks like the only real solution is the Skopeo update, which we're now doing without issue.

[stuartleeks]

@aaronadamsCA - thanks for the follow-up message

[m-roberts]

I was able to update the version of Skopeo on an Ubuntu 22.04 GitHub Actions runner with the following:

REPO_URL="https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/unstable/xUbuntu_22.04"
sudo sh -c "echo 'deb ${REPO_URL}/ /' > /etc/apt/sources.list.d/skopeo.list"
curl -fsSL ${REPO_URL}/Release.key | gpg --dearmor | sudo tee /etc/apt/trusted.gpg.d/skopeo.gpg > /dev/null

sudo apt update
sudo apt install skopeo

I find this preferable to the suggestion above by @natescherer as it uses fewer steps and is easier to read.

[lejouson]

Following advice from https://github.com/containers/skopeo/issues/1874#issuecomment-1541088511, I have got a successful build by setting BUILDX_NO_DEFAULT_ATTESTATIONS.

- name: Pre-build and push image
  uses: devcontainers/[email protected]
  env:
    BUILDX_NO_DEFAULT_ATTESTATIONS: true
  with:
    imageName: ghcr.io/${{ github.repository }}
    cacheFrom: ghcr.io/${{ github.repository }}
    platform: linux/amd64,linux/arm64
    push: always

[aaronadamsCA]

Very nice, that is much cleaner and faster. I can confirm this works for me as well.

Complete working multi-platform build workflow

on: push
jobs:
  devcontainer:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/[email protected]

      - uses: docker/[email protected]
        with:
          registry: ghcr.io
          username: "${{ github.repository_owner }}"
          password: "${{ github.token }}"

      - uses: docker/[email protected]
        with:
          platforms: linux/amd64,linux/arm64

      - uses: docker/[email protected]

      - uses: devcontainers/[email protected]
        env:
          BUILDX_NO_DEFAULT_ATTESTATIONS: true
        with:
          cacheFrom: "ghcr.io/${{ github.repository }}/devcontainer"
          imageName: "ghcr.io/${{ github.repository }}/devcontainer"
          platform: linux/amd64,linux/arm64
          refFilterForPush: refs/heads/main
          subFolder: .devcontainer

[onedr0p]

It appears that BUILDX_NO_DEFAULT_ATTESTATIONS workaround no longer works. I tried it and I am back to the tagging error again. I had to go back to @zydou method for it to work again.

[chrmarti]

Adding a fix for the tagging error in https://github.com/devcontainers/ci/issues/271.

I also had to update skopeo to get around the mime type error. Ubuntu 22.04 (the current LTS version) has skopeo 1.4.1, it seems GitHub's ubuntu-latest uses just that. I guess 24.04 might be the next LTS version and I see skopeo 1.9.3 in Ubuntu 23.10. 🤞