push failed: reason:changing manifest media-type from "application/vnd.docker.distribution.manifest.v2+json" to "application/vnd.docker.distribution.manifest.list.v2+json" is disallowed reference:master

[metacoma]

$ func version
v0.44.0
$ func create --language typescript
$ func build --image 10.24.142.129:30001/func:master --registry=10.24.142.129:30001 --registry insecure --push
....
Error: cannot write image index: PUT http://10.24.142.129:30001/v2/func:master: MANIFEST_INVALID: manifest invalid; map[description:During upload, manifests undergo several checks ensuring validity. If those checks fail, this error MAY be returned, unless a more specific error is included. The detail will contain information the failed validation. reason:changing manifest media-type from "application/vnd.docker.distribution.manifest.v2+json" to "application/vnd.docker.distribution.manifest.list.v2+json" is disallowed reference:master]

but

$ docker push 10.24.142.129:30001/func:master
The push refers to repository [10.24.142.129:30001/func:master]
83d85471d9f8: Layer already exists 
....
270a1170e7e3: Layer already exists 
master: digest: sha256:acd83416b3d9595ecbdc68a86452940da4ee9c939a41ad14e42de0c48fd99a05 size: 2834

[metacoma]

Just FYI, in-cluster-build works fine (single node kubernetes)

func deploy --build=false --image zot-int.zot.svc.cluster.local:5000/mindwm-function/mindwm-typescript:master --registry-insecure --remote -R
Running Pipeline Task: scaffold
Running Pipeline Task: Building function image on the cluster
Running Pipeline Task: Deploying function to the cluster
✅ Function updated in namespace "default" and exposed at URL: 
   http://mindwm-function.default.10.24.142.129.nip.io
Function Deployed at http://mindwm-function.default.10.24.142.129.nip.io

cat /etc/docker/daemon.json

{
  "hosts": ["unix:///var/run/docker.sock", "tcp://0.0.0.0:2375" ],
  "insecure-registries" : ["10.24.142.129:30001", "zot-int.zot:5000", "zot-int.zot.svc.cluster.local:5000", "10.43.99.225:5000"]
}

10.24.142.129:30001 and zot-int.zot.svc.cluster.local:5000 is the same docker registry

[matejvasek]

@jrangelramos have we ever observed this bug?

[matejvasek]

One thing that func does with push is that it pushes "multi-arch" image with a single architecture in it.

At surface this may not make much sense but this is done for a good reason. With this multi-arch image the pod is allocated at correct node. This is useful if the cluster happens to have nodes with varying architectures.

The on cluster build does push plain image not multi-arch image so it works. However we probably should change it to produce multi-arch too.

[matejvasek]

I suspect your registry does not like pushing multi-arch image to it for some reason.

[matejvasek]

You could try to push manifest (mult-arch) manually, something like:

docker manifest create  10.24.142.129:30001/func:latest \
  10.24.142.129:30001/func:amd64
docker manifest push 10.24.142.129:30001/func:latest

See: Approach 1: Without Using Docker BuildX or https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way "The hard way with docker manifest"

[matejvasek]

@metacoma what kind of registry do you use?

[metacoma]

@metacoma what kind of registry do you use?

Zot-registry 0.1.65 installed using helm with default values

[metacoma]

I suspect your registry does not like pushing multi-arch image to it for some reason.

Yes, probably https://github.com/project-zot/zot/issues/1411 related

[metacoma]

You could try to push manifest (mult-arch) manually, something like:

docker manifest create 10.24.142.129:30001/func:latest
10.24.142.129:30001/func:amd64 docker manifest push 10.24.142.129:30001/func:latest See: Approach 1: Without Using Docker BuildX or https://www.docker.com/blog/multi-arch-build-and-images-the-simple-way "The hard way with docker manifest"

Sorry, did I miss something?

+ func create --language typescript test-function2
Created typescript function in /tmp/mindwm-gitops/hacks/test-function2/test-function2/test-
function2
+ cd test-function2
+ REGISTRY_URL=10.24.142.129:30001
+ FUNC_IMAGE_REPO=10.24.142.129:30001/func
+ FUNC_IMAGE_TAG=master
+ FUNC_IMAGE=10.24.142.129:30001/func:master
+ func build --image 10.24.142.129:30001/func:master --registry=10.24.142.129:30001 --regis
try insecure
Building function image
Still building
🙌 Function built: 10.24.142.129:30001/func:master
+ docker manifest create 10.24.142.129:30001/func:master 10.24.142.129:30001/func:amd64
no such manifest: 10.24.142.129:30001/func:amd64

[matejvasek]

Interesting, I checked our code and it appears that we do not push under ${arch} tag. So what probably happens is more like

docker manifest create 10.24.142.129:30001/func:master 10.24.142.129:30001/func:master

and apparently zot has problem with that the tag is identical.

We never seen this issue on our registries.

[matejvasek]

We probably could try change the tag for the image included int the manifest.

[metacoma]

We probably could try change the tag for the image included int the manifest.

I don't fully understand what you mean, but if you have a quick fix or patch, I can try it out promptly.

and apparently zot has problem with that the tag is identical.

You can easily reproduce my issue:

# REGISTRY_IP=10.24.142.129 # <--- change
# REGISTRY_PORT=30001
helm repo add zot "https://zotregistry.dev/helm-charts/"
helm repo update
helm -n zot2 upgrade --version 0.1.65 --install --create-namespace zot zot/zot --set service.type=NodePort --set service.nodePort=${REGISTRY_PORT}
# add ${REGISTRY_IP}:${REGISTRY_PORT} to insecure registries in /etc/docker/daemon.json
# restart docker
func create --language typescript test-function2
cd test-function2

func build --image ${REGISTRY_IP}:${REGISTRY_PORT}/func:master --registry=${REGISTRY_IP}:${REGISTRY_PORT} --registry-insecure --push

[matejvasek]

@metacoma you could try:

diff --git a/pkg/docker/pusher.go b/pkg/docker/pusher.go
index 2f85e0c30..6fd578a90 100644
--- a/pkg/docker/pusher.go
+++ b/pkg/docker/pusher.go
@@ -190,6 +190,8 @@ func (n *Pusher) pushImage(ctx context.Context, f fn.Function, credentials Crede
 		return "", errors.New("Function has no associated image.  Has it been built?")
 	}
 
+	f.Build.Image += "-adhocsuffix"
+
 	registry, err := GetRegistry(f.Build.Image)
 	if err != nil {
 		return "", err

[matejvasek]

Oh it won't be as simple as I wrote above.

[matejvasek]

Another thing I noticed: I cannot push image tarball using crane with zot. I got:

Error: PUT http://localhost:8080/v2/ns2/fn-node/manifests/latest-xyz: MANIFEST_INVALID: manifest invalid; map[description:During upload, manifests undergo several checks ensuring validity. If those checks fail, this error MAY be returned, unless a more specific error is included. The detail will contain information the failed validation. mediaType:application/vnd.docker.distribution.manifest.v2+json]

[github-actions[bot]]

This issue is stale because it has been open for 90 days with no activity. It will automatically close after 30 more days of inactivity. Reopen the issue with /reopen. Mark the issue as fresh by adding the comment /remove-lifecycle stale.

[kbaegis]

I've hit this as well.