Instance doesn't get cleaned up with Kubernetes driver

[davidspek]

Behaviour

When using the Kubernetes driver the buildx builder instance isn't getting cleaned up, leaving dangling deployments on the cluster (which in our case is leaving extra nodes on the cluster indefinitely).

Steps to reproduce this issue

1. Use the kubernetes driver
2. Have a build run
3. Note the builder isn't cleaned up

Expected behaviour

The builder gets cleaned up removing the Kubernetes deployment

Actual behaviour

Kubernetes deployments of builder instances linger indefinitely.

Configuration

• Repository URL (if public): https://github.com/pluralsh/plural-cli/
• Build URL (if public): https://github.com/pluralsh/plural-cli/actions/runs/6161237552/job/16720454186#step:26:2

- name: Set up Docker Buildx
        id: builder
        uses: docker/setup-buildx-action@v3
        with:
          cleanup: true
          driver: kubernetes
          platforms: linux/amd64
          driver-opts: |
            namespace=buildx
            requests.cpu=1.5
            requests.memory=3.5Gi
            "nodeselector=plural.sh/scalingGroup=buildx-spot-x86"
            "tolerations=key=plural.sh/capacityType,value=SPOT,effect=NoSchedule;key=plural.sh/reserved,value=BUILDX,effect=NoSchedule"
      - name: Append ARM buildx builder from AWS
        run: |
          docker buildx create \
            --append \
            --bootstrap \
            --name ${{ steps.builder.outputs.name }} \
            --driver=kubernetes \
            --platform linux/arm64 \
            --node=${{ steps.builder.outputs.name }}-arm64 \
            --buildkitd-flags "--allow-insecure-entitlement security.insecure --allow-insecure-entitlement network.host" \
            --driver-opt namespace=buildx \
            --driver-opt requests.cpu=1.5 \
            --driver-opt requests.memory=3.5Gi \
            '--driver-opt="nodeselector=plural.sh/scalingGroup=buildx-spot-arm64"' \
            '--driver-opt="tolerations=key=plural.sh/capacityType,value=SPOT,effect=NoSchedule;key=plural.sh/reserved,value=BUILDX,effect=NoSchedule"'

Logs

Download the log file of your build and attach it to this issue.

[crazy-max]

I think the current API for the Kubernetes driver in Buildx is using the pod.spec. For this case we would need a driver option to use ephemeralcontainers maybe? Any idea @AkihiroSuda @morlay?