Document how to configure a registry to allow deleting images

[gouyang]

# skopeo  delete --tls-verify=false docker://localhost:5000/skopeo/busybox:latest
FATA[0000] Failed to delete /v2/skopeo/busybox/manifests/sha256:4a6e0504c306bdb5425bc8efe54c5ab7e8ede5a5ebe031f4907cb081b05f3a38: {"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
 (405 Method Not Allowed) 

[gouyang]

# ./skopeo --debug  delete --tls-verify=false docker://localhost:5000/skopeo/busybox:latest
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration 
DEBU[0000]  Using "default-docker" configuration        
DEBU[0000]   Using file:///var/lib/atomic/sigstore      
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/localhost:5000 
DEBU[0000] GET https://localhost:5000/v2/               
DEBU[0000] Ping https://localhost:5000/v2/ err &url.Error{Op:"Get", URL:"https://localhost:5000/v2/", Err:(*errors.errorString)(0xc42049c8a0)} 
DEBU[0000] GET http://localhost:5000/v2/                
DEBU[0000] Ping http://localhost:5000/v2/ err <nil>     
DEBU[0000] Ping http://localhost:5000/v2/ status 200    
DEBU[0000] GET http://localhost:5000/v2/skopeo/busybox/manifests/latest 
DEBU[0000] DELETE http://localhost:5000/v2/skopeo/busybox/manifests/sha256:3fd16eb81fa406aacc9579876e3fba5d9446e9e1f25475a9519ca7a36525696c 
FATA[0000] Failed to delete /v2/skopeo/busybox/manifests/sha256:3fd16eb81fa406aacc9579876e3fba5d9446e9e1f25475a9519ca7a36525696c: {"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
 (405 Method Not Allowed) 

[mtrmac]

Is the registry configured to allow image deletion, as documented in https://github.com/projectatomic/skopeo/blob/master/docs/skopeo.1.md#skopeo-delete ?

[gouyang]

After execute the container registry garbage collector, it still not work.

[root@fedora bin]# /usr/bin/registry garbage-collect /etc/docker-distribution/registry/config.yml
buildah/busybox
buildah/busybox: marking manifest sha256:34ca4978e48181b47028c47ab5254c7bbb0446437383fc09c759311b3e3480c9 
buildah/busybox: marking blob sha256:2c9f2c72031599f0a2a0076b4cfc4474628a4f90652501ed9b4f58e220882580
buildah/busybox: marking blob sha256:0ffadd58f2a61468f527cc4f0fc45272ee4a1a428abe014546c89de2aa6a0eb5
skopeo/busybox
skopeo/busybox: marking manifest sha256:3fd16eb81fa406aacc9579876e3fba5d9446e9e1f25475a9519ca7a36525696c 
skopeo/busybox: marking blob sha256:6ad733544a6317992a6fac4eb19fe1df577d4dec7529efec28a5bd0edad0fd30
skopeo/busybox: marking blob sha256:0ffadd58f2a61468f527cc4f0fc45272ee4a1a428abe014546c89de2aa6a0eb5
skopeo/busybox: marking manifest sha256:4a6e0504c306bdb5425bc8efe54c5ab7e8ede5a5ebe031f4907cb081b05f3a38 
skopeo/busybox: marking blob sha256:54511612f1c4d97e93430fc3d5dc2f05dfbe8fb7e6259b7351deeca95eaf2971
skopeo/busybox: marking blob sha256:03b1be98f3f9b05cb57782a3a71a44aaf6ec695de5f4f8e6c1058cd42f04953e
skopeo/nginx
skopeo/nginx: marking manifest sha256:d285d1c14ed35681414520f86c785931cb635ed8fafe32ca340ff07a67f9bf22 
skopeo/nginx: marking blob sha256:e54095500ebea705b023dd2a6e7444be3d0d7534cd092348d40dcb4bc958b895
skopeo/nginx: marking blob sha256:bc95e04b23c06ba1b9bf092d07d1493177b218e0340bd2ed49dac351c1e34313
skopeo/nginx: marking blob sha256:f3186e650f4e1edfa976c4a930470e8a75f6c40fd8239d05abf7a9a9b13a2896
skopeo/nginx: marking blob sha256:9ac7d662170839889c79754e924517ec602076e7500fbf2126a47a8513dbe3e3
skopeo/nginx: marking blob sha256:4f4fb700ef54461cfa02571ae0db9a0dc1e0cdb5577484a6d75e68dc38e8acc1

14 blobs marked, 0 blobs eligible for deletion

[root@fedora bin]# systemctl restart docker-distribution

[root@fedora bin]# skopeo delete --tls-verify=false docker://localhost:5000/skopeo/nginx:latest
FATA[0000] Failed to delete /v2/skopeo/nginx/manifests/sha256:d285d1c14ed35681414520f86c785931cb635ed8fafe32ca340ff07a67f9bf22: {"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
 (405 Method Not Allowed) 

[mtrmac]

I meant configuring the server in advance, not after-the-fact garbage cleanup. The man page says REGISTRY_STORAGE_DELETE_ENABLED=true, I guess in the environment of the registry server; the same configuration can probably be set up using the config file of the registry.

[CamiloGarciaLaRotta]

Hi, to avoid duplication I am appending here as this issue as it is partly correlated to mine:

I am able to delete images manually from out private Artifactory registry. I am able to inspect and copy images in that registry through Skopeo.

Unfortunately, I am unable to delete images from the registry, I obtain the following error:

$ skopeo --debug delete docker://<REGISTRY>/<IMAGE>:<VERSION> --creds=<U>:<P>
DEBU[0000] Using registries.d directory /etc/containers/registries.d for sigstore configuration
DEBU[0000]  Using "default-docker" configuration
DEBU[0000]   Using file:///var/lib/atomic/sigstore
DEBU[0000] Looking for TLS certificates and private keys in /etc/docker/certs.d/<REGISTRY>
DEBU[0000] GET https://<REGISTRY>/v2/
DEBU[0000] Ping https://<REGISTRY>/v2/ err <nil>
DEBU[0000] Ping https://<REGISTRY>/v2/ status 401
DEBU[0001] GET https://<REGISTRY>/v2/<IMAGE>/manifests/<VERSION>
DEBU[0002] DELETE https://<REGISTRY>/v2/<IMAGE>/manifests/sha256:<SHA>
FATA[0003] Failed to delete /v2/<IMAGE>/manifests/sha256:<SHA>: {
  "errors" : [ {
    "status" : 406,
    "message" : "Not Acceptable"
  } ]
} (406 Not Acceptable)

As a side note, in your skopeo delete documentation you use the --force flag which when I run it appears to not be supported anymore:

$ skopeo --debug delete docker://<REGISTRY>/<IMAGE>:<VERSION> --creds=<U>:<P>
Incorrect Usage.
FATA[0000] flag provided but not defined: -force

[gouyang]

The problem is not existed if add delete: enabled: true to the config file:

cat /etc/docker-distribution/registry/config.yml 
version: 0.1
log:
  fields:
    service: registry
storage:
    cache:
        layerinfo: inmemory
    filesystem:
        rootdirectory: /var/lib/registry
    delete:
        enabled: true
http:
    addr: :5000

Below steps show that it delete the image successfully.

$ rpm -q skopeo
skopeo-0.1.31-5.git0144aa8.fc28.x86_64
$ sudo skopeo inspect --tls-verify=false docker://localhost:5000/skopeo/busybox:latest
{
    "Name": "localhost:5000/skopeo/busybox",
    "Digest": "sha256:a9d04077e7698059d282d9e89a1676a045b2bd0b33d059b1dde77e80e613b5d3",
    "RepoTags": [
        "latest"
    ],
    "Created": "2018-05-23T21:19:31.132152818Z",
    "DockerVersion": "17.06.2-ce",
    "Labels": null,
    "Architecture": "amd64",
    "Os": "linux",
    "Layers": [
        "sha256:07a152489297fc2bca20be96fab3527ceac5668328a30fd543a160cd689ee548"
    ]
}
$ sudo skopeo delete --tls-verify=false docker://localhost:5000/skopeo/busybox:latest
$ sudo skopeo inspect --tls-verify=false docker://localhost:5000/skopeo/busybox:latest
FATA[0000] Error reading manifest latest in localhost:5000/skopeo/busybox: manifest unknown: manifest unknown 
```

[rhatdan]

@TomSweeneyRedHat or @gouyang would you guys like to write up a quick blog on how to set this up and we can publish it, and then add it to the skopeo readmes.

[gouyang]

@TomSweeneyRedHat Would you please do this, the reference doc is: https://docs.docker.com/registry/configuration/#delete

[rhatdan]

@TomSweeneyRedHat Reminder, Did you ever write the blog?

[TomSweeneyRedHat]

@rhatdan I did not and this completely slipped off my radar. WIll get it back on the list.

[irispenker]

Hi I am running Skopeo from a container . Managed to copy images from one registry to another, which is AWS ECR, no need to special configuration . Now i am trying to delete an image from same ECR, and i get same errors. I tried deleting with digest and is says my user name password is wrong , same user can simply delete the image from elsewhere.

Can someone help by saying :

1)Should i have some registry yml configuration set for my ecr just for delete ?

2)If so - where should it be located ? so many different locations are mentioned in different posts : According to https://github.com/containers/skopeo/blob/master/docs/skopeo.1.md#skopeo-delete config should be here : /etc/containers/storage.conf But according to https://docs.docker.com/registry/configuration/#delete Docker is expecting in other location. And the above mention : /etc/docker-distribution/registry/config.yml What should it be ???

3)If i do need this file , how should it look like for AWS ECR ??

I must mention that i am not using docker directly within the container - only Skopeo. So if you go to my container and type docker command it will error that "docker not found" .

Can someone help ?

[mtrmac]

(The AWS/ECR question was discussed separately in #950 . Leaving this to track a blog post about enabling deletion in registries.)

[rhatdan]

@QiWang19 Could you throw together a blog to describe this?

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.

[willzhang]

2021年7月15日，can not do it still:

[root@master ~]# skopeo delete docker://willdockerhub/nginx:latest
FATA[0004] Failed to delete /v2/willdockerhub/nginx/manifests/sha256:794275d96b4ab96eeb954728a7bf11156570e8372ecd5ed0cbc7280313a27d19: {"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
 (405 Method Not Allowed) 
[root@master ~]# 

[vrothberg]

Ping @TomSweeneyRedHat & @lsm5

[lsm5]

ack, this fell through the cracks. I'll check with Tom and get this going soon.

[github-actions[bot]]

A friendly reminder that this issue had no activity for 30 days.