Delete images (repository + data) from registry

[sourav82]

What are the ways to delete the pushed images and repositories from registry?

I guess, Docker does not provide any API to do that operation. Any ways, may be using curl, to delete image data from registry?

[squarism]

I don't have an answer but have a similar question.

I pushed the busybox image to my private 2.0 registry (distribution). This will work for me. I get a huge json payload.

curl -XGET http://private-host:5000/v2/busybox/manifests/latest
# GET included for effect only  :)

The API doc says that DELETE should work. So I just substitute above and get

curl -XDELETE http://private-host:5000/v2/busybox/manifests/latest                                                                                            
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

So I think that's your answer but it's not working for me.

[dmp42]

Delete will remove the tag, but not the actual payloads. So, your image will no longer be available, it will still exist on your backend storage.

[dmp42]

Some people have contributed garbage-collector scripts to do the cleanup. You will find them clicking the "delete" label over here.

[squarism]

I think the issue here is that I'm getting a method not supported on the DELETE request not that the blobs are sticking around. If you have an example curl example, that's be great. I could be doing something wrong here. If the OP disagrees, I'll open another issue.

[dmp42]

@squarism so, you are using the golang registry2.

Please head over to https://github.com/docker/distribution and/or irc #docker-distribution

TLDR: delete is not supported yet, in design phase: https://github.com/docker/distribution/issues/210

[squarism]

Aaah! And I blogged the url to the distribution repo too! I'm sorry, I knew the difference, I failed to see what issue list I was in. :sob: Thanks for the tldr too. :+1:

[dmp42]

;)

[alwqx]

+1 with my operation and output:

adolph@geek:docker_registry$ curl -v -X DELETE http://localhost:5000/v2/busybox/manifests/sha256:0fc02bc170932a0d727de2201c65bfe3fe31448dfed6f6bcbc1b37c2668927f5
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> DELETE /v2/busybox/manifests/sha256:0fc02bc170932a0d727de2201c65bfe3fe31448dfed6f6bcbc1b37c2668927f5 HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: */*
> 
< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Dec 2015 05:57:29 GMT
< Content-Length: 78
< 
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
* Connection #0 to host localhost left intact
adolph@geek:docker_registry$ curl -v -X DELETE http://localhost:5000/v2/busybox/manifests/list
* Hostname was NOT found in DNS cache
*   Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> DELETE /v2/busybox/manifests/list HTTP/1.1
> User-Agent: curl/7.35.0
> Host: localhost:5000
> Accept: */*
> 
< HTTP/1.1 405 Method Not Allowed
< Content-Type: application/json; charset=utf-8
< Docker-Distribution-Api-Version: registry/2.0
< X-Content-Type-Options: nosniff
< Date: Fri, 25 Dec 2015 05:59:35 GMT
< Content-Length: 78
< 
{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}
* Connection #0 to host localhost left intact

[CodeJuan]

I sent the same request with @adolphlwq 's request, and got the same response

curl -v -X DELETE http://myregistry/v2/busybox/manifests/sha256:blablabla...

{"errors":[{"code":"UNSUPPORTED","message":"The operation is unsupported."}]}

update

I got the solution to delete images

enable delete

set the environment variable REGISTRY_STORAGE_DELETE_ENABLED = True

the API to delete image

1. get the manifest from registry

get v2/<repoName>/manifests/<tagName>

the Docker-Content-Digest is response.Header["Docker-Content-Digest"] the layerDigests is response.body["fsLayers"]["blobSum"]

1. delete layerDigests

delete v2/<repoName>/blobs/<layerDigests>

1. delete Docker-Content-Digest

delete v2/<repoName>/manifests/<Docker-Content-Digest>

1. then pull the image from registry, the response is invalid character '<' looking for beginning of value

But when I get 'v2/repoName/tags/list', the tag which was been deleted is still exist.......

[mphanikumars]

Hi,

I am very new to Docker.

I have tried like this "curl -X DELETE localhost:5000/v1/repositories/jenkins_ora_jdk/tags/latest" but this works for tag deletion and not the image data for that tag. So i had to write my own script to do that Job. Since i am only the one who tested this script please use with your own risk

This is how i start registry image mkdir -p /docker/registry/latest && docker run -d -p 5000:5000 -v /docker/registry/latest:/tmp registry

To delete an Image with tag : (This will also cleans the data specific to that tag)

./docker_registry_delete_image -d 5000 "jenkins_ora_jdk:latest"

To see all the Images with tag :

./docker_registry_delete_image -q 5000

: 5000 is Registry PORT

# NOTE

1. Registry service must be running

2. Volume must be mounted locally ( -v /docker/registry/latest:/tmp )

3. This works only in Linux environment

4. This should be executed in where docker registry is running

docker_registry_delete_image.sh.txt

Thanks, Phani Kumar

[lioncui]

@CodeJuan You must edit registry config.xml with

storage:
    delete:
        enabled: true

reference https://docs.docker.com/registry/spec/api/

[slartibart70]

Hi, just adding my experiences:

• started registry:2 with my own config (deleting allowed) and own volume mounted storage.
• Pushing images into my registry works ok, my storage contains the data (approx. 100MB)

Now trying to delete:

• curl -v -s -X DELETE "localhost:5000/v2/collctor/blobs/sha256:cbb2acf4..." (see layerDigests above)
• curl -v -s -X DELETE "localhost:5000/v2/collctor/manifests/sha256:66846..." (see Docker-Content-Digest above)

After the above steps, we still have an entry in the catalog: curl -v -s -X GET "localhost:5000/v2/_catalog" .. which still returns the deleted image curl -s -X GET localhost:5000/v2//manifests/latest still returns seemingly valid data but a docker pull localhost:5000/ does not work any more. And, the storage space (my ~100MB) are still in use and are not deleted.

Any ideas?

[dmp42]

This repo here is deprecated, and any message here is likely to go unnoticed. Either report problems on the docker/distribution repo, or on irc (#docker-distribution)

[EugenMayer]

I finally go teached about storage->delete->true in the configuration https://github.com/docker/distribution/blob/master/docs/configuration.md

and implemented it. Still, while fetching the digest https://github.com/EugenMayer/docker_registry_cli/blob/master/DockerRegistryRequest.rb#L92 getting a string like sha256:XXXX

Using this to feed into delete https://github.com/EugenMayer/docker_registry_cli/blob/master/DockerRegistryRequest.rb#L83 i get notfied, that the digest is wrong.

Using the current latest registry:2 (sha256:20f5d95004b71fe14dbe7468eff33f18ee7fa52502423c5d107d4fb0abb05c1d).

Is this a bug / fixed already?

[jshapiro26]

@EugenMayer I've been messing around with this too. It looks like the digest that you return as a header from: get v2/<repoName>/manifests/<tagName> is the wrong digest.

Instead, if you docker pull that image from your registry and grab the Digest it spits out after downloading you should should be able to delete using that with:

curl -u <username:password> -X DELETE https://yourregistry/v2/<repository>/manifests/<newdigest>

If anyone knows of a simple way to get this digest other than pulling the image, please let me know.

Once that is deleted, run garbage collection dry run and then garbage collect on your registry to actually delete the data that you have marked for deletion. On the docker host that is running your registry, run: docker exec -it <name of container or container ID> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml. It shows that you have N blobs eligible for deletion. Run the command again without --dry-run and it will delete the blobs. Now that they're deleted, if you run the dry-run again you should see that the namespace you deleted has no more blobs.

[EugenMayer]

as requested i created a new issue #1068, maybe you continue work with me there @jshapiro26

[EugenMayer]

ups, used the legacy issue, @jshapiro26 see https://github.com/docker/distribution/issues/1637 instead

[sergey-tkalych]

My approach how to delete image tag from docker registry (tested on registry version 2.4.0):

1) Update registry configuration file (config.yml) and restart registry

storage:
    delete:
        enabled: true

2) Get tags list

GET ://<registry_host>/v2/<repo_name>/tags/list

3) Get manifest for selected tag

GET ://<registry_host>/v2/<repo_name>/manifests/<tag_name>

4) Copy digest hash from response header

Docker-Content-Digest: <digest_hash>

5) Delete manifest (soft delete). This request only marks image tag as deleted and doesn't delete files from file system. If you want to delete data from file system, run this step and go to the next step

DELETE ://<registry_host>/v2/<repo_name>/manifests/<digest_hash>

Note! You must set headers for request - Accept: application/vnd.docker.distribution.manifest.v2+json

6) Delete image data from file system

Run command from the host machine:

docker exec -it <registry_container_id> bin/registry garbage-collect <path_to_registry_config>

Note! Usually, <path_to_registry_config>=/etc/docker/registry/config.yml

[EugenMayer]

Yes, i already implemented this in my docker_registry_cli: https://github.com/EugenMayer/docker_registry_cli/blob/master/commands/DockerRegistryCommand.rb#L57

So this works - one of the things i missed was the Accept header: application/vnd.docker.distribution.manifest.v2+json]

But other then that, it works quiet butifully

Hint: dont forget to run the garbage collector to clean up the space

[lvthillo]

There is no API available to trigger the GC so I think it's the only possibility to use docker exec .. ? I'm able to put it in a cron job or something but I don't know how I can get my registry in read-only mode, execute it and disable read-only (we need to push images to the registry, but it has to be forbidden when the GC runs). Or is the only solution to stop the container, start it as read-only, docker exec .., stop container and start it in a normal way? This would take "a lot of" time I assume. (I want to run the GC every time we delete an image)

[EugenMayer]

For, the GC is utterly broken in 2.5 - i would assume the registry being completely useless after more then 2 months - you have to wipe it a redo all images. There are severe issues with blobs, pushed will not work because "the layers are already up to date". Using the GC is completely useless, the only thing actually really kinda works is going to /var/lib/registry/repos.. and delete the folder manually - restart the registry container.

[jshapiro26]

@lorenzvth7 and @EugenMayer, I ended up writing an interactive script for the process I mentioned above and will share it soon and post back here, maybe it'll be useful for someone. The issue i've had is that my team typically just uses the latest tag. The latest tag then starts bloating and there isn't a good way to clean up old versions of latest.

I ended up just parsing the output of the --dry-run garbage collection command and comparing the SHAs against the SHA I get from latest when I run a docker pull. This is because Docker-Content-Digest: <digest_hash> never seems to be correct for the DELETE call to the API.

I noticed your comment about the header: Accept header: application/vnd.docker.distribution.manifest.v2+json and will check that out. Maybe it'll solve all the problems!

[lvthillo]

@jshapiro26 I'm able to get the right sha of my image with tag:latest. I'm using this command:

curl -k -v --silent -H "Accept: application/vnd.docker.distribution.manifest.v2+json" -X GET https://myreg/v2/proj2/base/manifests/latest 2>&1 | grep Docker-Content-Digest | awk '{print ($3)}'
sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe

Check in my registry

$ docker exec -it 86deeac4f25f /bin/sh
/ # cd /var/lib/registry/docker/registry/v2/repositories/proj2/base/_manifests/revisions/sha256/52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe/

Hope it helps for your. My issue is the following now: I delete the sha: curl -k -v --silent -X DELETE https://myreg/v2/proj2/base/manifests/sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe

This is going fine:

> Accept: */*
>
< HTTP/1.1 202 Accepted
< Docker-Distribution-Api-Version: registry/2.0

I use the GC to clean up:

docker exec -it 86deeac4f25f bin/registry garbage-collect  /etc/docker/registry/config.yml
$ docker exec -it 86deeac4f25f bin/registry garbage-collect /etc/docker/registry/config.yml
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/c8/c8144262002cd241e607d7d3ecda450ce4ae8edf7dac8dbf46897d498ac667d8  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/ce/cee0974db2b868f0408f7e3eaba93c11fce3a38f612674477653b04c10369da0  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/06/064f9af025390d8da3dfab763fac261dd67f8807343613239d66304cda8f5d16  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/39/390957b2f4f0cd72b8577795cd8076cdc21d45c7823bbb5c895a494ae6038267  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/3d/3d1d7dffe2da492324b1fd62a2939ddc6e50549746cefa9a67684253dfa40b88  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/52/52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/6b/6bc128ff8a43e6b26862384494bd2cbfabc5e9d5277121881cbd0fcbf1442508  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392
INFO[0000] Deleting blob: /docker/registry/v2/blobs/sha256/a5/a52d570eb492f9fb3cebb345c96395fc6156eef00ac42758e1a2d0f4190ecbf6  go.version=go1.6.3 instance.id=5ba1a41c-a0a4-4f87-a3c0-0cd1a1a2d392

Check (seems good to me):

/var/lib/registry/docker/registry/v2 # du -sh
63.0K

But now the real issue is repushing:

The push refers to a repository [myreg/proj2/base]
5d3accbaa61c: Layer already exists
3af563f9cad6: Layer already exists
447f88c8358f: Layer already exists
df9a135a6949: Layer already exists
dbaa8ea1faf9: Layer already exists
8a14f84e5837: Layer already exists
latest: digest: sha256:52cfbde9c62b2df9a4fa6f06e56759431166d2374bb0812cf0d6ee781cbe1abe size: 1576

Everything seems to exist already in the registry (isn't it cleaned up?). The sha is the same which seems normal to me (it's the same image). But checking in the registry?

/var/lib/registry/docker/registry/v2 # du -sh
65.0K

The most of space is taken in /bin/registry

/bin # du -sh registry
26.0M   registry

Something I don't understand. after deleting the sha and rerunning the GC nothing seems to happen (the tag and _manifest is gone in the registry, but it seems not necessary to delete layers again, the space in / remains the same?).

[jshapiro26]

Thanks for the clarification about sending the header! I'll try that out. It sounds like you likely have an older version of that image that would be untagged now since you've deleted latest. If you remove that image again then ssh to your registry and run docker exec -it <container ID or name> bin/registry garbage-collect --dry-run /etc/docker/registry/config.yml does it return any SHAs or blobs under myreg/proj2/base? That might be where the "Layer already exists" is coming from as lots of layers may have been unchanged.

The script i'm cleaning up goes through and deletes all old SHAs for a specified repo, leaving the one tagged as latest in-tact.

[lvthillo]

I pushed the image only with the latest tag. Than I'm able to delete it correctly but after repushing just the same image with just the same tag it goes wrong. Everything seems to exist, All the folders in /var/lib/registry/docker/registry/v2/blobs/sha256/xx are empty. There is a new tag and a new manifest in the registry but it does not take any space. Even pulling the image from the registry does not work..

When I push another image with the same name and same tag (also latest in this case) it's pushing in a fine way (I build it with --no-cache, don't know if it's relevant). I'm able to pull it and everything so this issue only seems to appear when I try to push, delete, repush and pull the exactly same image.

Maybe it's all a bit unclear but to resume: It seems to work fine for me (pushing, pulling, deleting, clearing) in any case except when I delete image:x and repush image:x (which is the same image, not only the same name). After the "repush" everything still seems to exist but I'm unable to pull.

[jshapiro26]

did you check the output of the --dry-run of garbage collection?

@EugenMayer here is the script I wrote to actually "garbage collect" https://github.com/jshapiro26/registry-tools/blob/master/clean_docker_registry.rb It's a bit of a hassle as it requires user input, but much faster than other methods i've seen.

I've set the garbage-collect task to run via cron on the docker host that my registry runs on, this script will mark the appropriate SHAs and blobs to be deleted when that cron runs.

[zoobab]

@EugenMayer I am running the same issue as yours with the "being unable to repush an image", as the deleted layers tells it is already there. When I list the images, it does not appear neither.

I have a similar shell script, will try to find out the --dry-run trick if it solves my repush issue.

[zoobab]

@jshapiro26 I am tempted to rewrite your ruby script in shell.

[zoobab]

@jshapiro26 your ruby script suffers from the same defect as @EugenMayer mentioned. When I repush the same image, I get the same message "Image already exists", and when I try to pull it from the repo, I got the error "Error: image ubuntu not found".

[jshapiro26]

@zoobab What image tag of registry are you using? I am using registry:2.4.1 and have never ran into the issue you're referring to. Also, do you run the garbage collection without --dry-run after you use the script and before pushing again? The script doesn't actually run the garbage collection after all the images and blobs are marked. I solve this by have a cron run the garbage collection from the docker host early in the morning everyday.

To clarify what I wrote the script for, it is used to remove all versions of an image that are not tagged "latest". Therefore if you're pushing an image with a tag that's called "6.2" or something like that, those images will be removed in addition to all the untagged versions of "latest" leaving only the image with the latest tag for the repo you're "cleaning".

I am unable to replicate what you're seeing @zoobab with this script. I use it daily and have an automated build of new images weekly that push successfully. Can you give me some more details so that I might be able to help?