problem deleting image or tag 16.6.1

[Gabrielandre02]

Whenever I try to delete an image or a tag, I'm getting this error but the logs in the registry aren't showing anything, nor are those in Gitlab. This installation was via a Linux package, I made the backup and left it in a new configuration with Docker Swarm, using Traefik as a reverse proxy

LOGS GITLAB:

gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,401 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,402 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,402 INFO Included extra file "/etc/supervisor/conf.d/gitlab-pages.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,402 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,402 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,402 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,403 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,403 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,403 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,403 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,403 INFO Set uid to user 0 succeeded
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,413 INFO RPC interface 'supervisor' initialized
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:07,413 INFO supervisord started with pid 1
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,418 INFO spawned: 'gitaly' with pid 897
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,423 INFO spawned: 'puma' with pid 898
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,428 INFO spawned: 'gitlab-workhorse' with pid 899
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,446 INFO spawned: 'sidekiq' with pid 900
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,451 INFO spawned: 'gitlab-pages' with pid 901
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,476 INFO spawned: 'sshd' with pid 907
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,504 INFO spawned: 'nginx' with pid 908
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:08,517 INFO spawned: 'cron' with pid 909
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,595 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,596 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,596 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,597 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,598 INFO success: gitlab-pages entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,598 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,599 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:49:09,599 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 09:55:20,380 INFO reaped unknown pid 1052
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 10:05:54,121 INFO reaped unknown pid 1084
gitlab_gitlab.1.6ovng6s6zitt@SRVDOCKER02    | 2023-12-14 10:09:14,909 INFO reaped unknown pid 1190

LOGS REGISTRY:

{"auth.user.name":"","err.code":"digest invalid","err.message":"provided digest did not match uploaded content","go.version":"go1.20.8","http.request.host":"registry_registry:5000","http.request.id":"09c7cdce-048b-4c9d-89e0-c368efc1b13f","http.request.method":"DELETE","http.request.remoteaddr":"10.0.1.232:48710","http.request.uri":"/v2/compmon/application/panorama_tunnel_ssh/nginx/manifests/alpine","http.request.useragent":"GitLab/16.6.1","http.response.contenttype":"application/json; charset=utf-8","http.response.duration":"7.641504ms","http.response.status":400,"http.response.written":98,"level":"error","msg":"response completed with error","time":"2023-12-15T12:18:36.491871799-03:00","vars.name":"compmon/application/panorama_tunnel_ssh/nginx","vars.reference":"alpine"}
registry_registry.1.og01zgrjx0v9@SRVDOCKER02    | 10.0.1.232 - - [15/Dec/2023:12:18:36 -0300] "DELETE /v2/compmon/application/panorama_tunnel_ssh/nginx/manifests/alpine HTTP/1.1" 400 98 "" "GitLab/16.6.1"

application_json.log

{"severity":"INFO","time":"2023-12-15T15:17:23.240Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"5d8fd23d3c5c78765a575cda99fdafd7","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","container_repository_id":85,"container_repository_path":"compmon/application/panorama_tunnel_ssh/nginx","project_id":189,"third_party_cleanup_tags_service":true}
{"severity":"ERROR","time":"2023-12-15T15:17:23.298Z","meta.caller_id":"ContainerRegistry::DeleteContainerRepositoryWorker","correlation_id":"5d8fd23d3c5c78765a575cda99fdafd7","meta.root_caller_id":"Cronjob","meta.feature_category":"container_registry","meta.client_id":"ip/","service_class":"Projects::ContainerRepository::DeleteTagsService","container_repository_id":85,"project_id":189,"message":"could not delete tags: alpine"}

[ymazzer]

Hi,

We are facing the same issue. In addition to the reported details, the gitlab registry cleanup policy when enabled will generate plenty of logs in both sides due to images not being deleted.

We got several instances, but the one we are facing this issue in is using registry 2.8.x instead of 2.7.x.

It seems related to several things :

• Gitlab has deprecated external registry support in 16.x

• Such an issue has been reported before

  • Gitlab uses image deletion by manifest which needs to be enabled on the registry
  • But it seems something has changed between registry 2.7.x and 2.8.x since gitlab uses an url like /v2/<repository>/manifests/<tag> in 16.6.2 and /v2/<repository>/manifests/<digest without sha256 prefix> making the deletion fail

  Is there any documented way to use gitlab registry with sameersbn image ?

[sachilles]

Hi,

unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry.

However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image.

Did anyone try this?

[ymazzer]

Hi @sachilles,

unfortunately I've been using only the integration of the docker registry into GitLab as pointed out in https://github.com/sameersbn/docker-gitlab/blob/master/docs/container_registry.md#gitlab-container-registry.

This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?

However, following https://docs.gitlab.com/ee/administration/packages/container_registry.html#self-compiled-installations, the integrated registry could be activated by using another image.

Did anyone try this?

I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\

[avvertix]

I'm also experiencing errors when deleting images from the container registry. I noticed the problem after upgrading to 16.6. I suppose it is connected to Gitlab forking the Docker distribution registry. In my case I'm currently using registry:2.7.1 from Docker Distribution to host container registry images. @ymazzer @sachilles are you using the same version?

[avvertix]

I actually stumbled on https://gitlab.com/gitlab-org/container-registry/-/issues/958#note_1471217687 that consider my setup (using registry:2.7.1) as a third party registry that will not be supported in the future.

In that thread is suggested to switch to the gitlab-container-registry version as declared in the Omnibus package. Apparently for Gitlab 16.6.2 the gitlab-container-registry version is v3.86.1-gitlab.

I tried the switch

1. I made a copy of the configuration file /etc/docker/registry/config.yml (apparently without an initial configuration file it won't start)
2. Replaced the registry docker image and mounted the new configuration file

   registry:
-    image: registry:2.7.1
+    image: "registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.86.1-gitlab"
     volumes:
+      - ./registry:/etc/docker/registry
       - /data/gitlab/registry:/registry
       - ./certs:/certs

I experienced an issue with file system permission during deletion, but I haven't figured out the correct user to switch to. I indeed temporary solved it by allowing write to everyone.

[sachilles]

Hi @ymazzer,

This is what we did, didn't you met the issue we are facing? Did you try to delete container images from gitlab interface or using the cleanup policy?

I have the same problem. It doesn't matter if I try to delete container images via the GitLab web interface or via the cleanup policy. The result is the same. (I'm using the "official" self-hosted docker registry in version 2.8.1.)

I didn't have time to try this out yet, I wanted to try using the integrated gitlab registry, directly from the main gitlab image by passing the registry disk to the main service, but I'm not sure it will work out of the box :\

Okay, I guess the proposed way is to migrate the content of the "official" self-hosted docker registry (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459).

[sachilles]

Dear @avvertix,

I found somewhere (see https://gitlab.com/gitlab-org/gitlab/-/issues/423459) that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry, as several changes were included after the container registry was forked by the GitLab maintainers. (My first attempt to replace the Docker registry with the GitLab container registry failed).

However, please make sure to backup the entire Docker registry data directories before the necessary migration.

Once I (or someone else) is successful, we will update the corresponding configuration files and documentation.

[Gabrielandre02]

@avvertix, This solution solved the problem and related to deleting the images. Tested on version 16.7.0. however I had to clean all the images within the registry.

Thanks

[avvertix]

https://gitlab.com/gitlab-org/gitlab/-/issues/423459 that the suggested way is to migrate the entire contents of the Docker registry by using the GitLab container registry

Dear @sachilles I also saw the link you shared, but my interpretation is that the issue refers to the migration from filesystem metadata (or object storage) to the database storage for the images metadata. As far as I understood migrating to the database metadata is in beta and not yet fully complete that's why I didn't try that way.

I saw a workaround following the feedback issue on the use of external registries.

I totally agree that my approach is just a workaround, but probably could make easier the migration path until all official migrations tools are available.

[avvertix]

however I had to clean all the images within the registry.

@Gabrielandre02 could you tell me more about why you had to clean all the images, as in the end I was able to use the same storage directory by ensuring that the user within the docker image had access to the filesystem. (btw seems that in my case the user on the host os didn't have write permission on a parent folder in the storage, after I ensured that all the folders/files are writeable by the current user everything worked without changing nothing)

[ymazzer]

Hi @sachilles @avvertix,

I just tried out the simple following procedure:

• Starting from a registry:2.8, I replaced the image by the latest gitlab-registry registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
• I changed registry mounted folder ownership to 1000:1000 and then started the registry and tried building an OCI image.
• Everything worked like a charm, I can now push/pull images from images and from my dev box as well as delete images from the gitlab interface or sidekiq jobs.

# docker-compose.yml
# [...]
  registry:
    #image: registry:2.8
    image: registry.gitlab.com/gitlab-org/build/cng/gitlab-container-registry:v3.88.0-gitlab
    restart: unless-stopped
    expose:
      - "5000"
    ports:
      - "127.0.0.1:5000:5000"
    volumes:
      - ./registry-config.yml:/etc/docker/registry/config.yml # this has changed as described before by @avvertix 
      - /opt/gitlab/disk/data/shared/registry:/var/lib/registry/docker/registry
      - /etc/ssl/private/registry.my-registry.com:/certs
    environment:
      REGISTRY_LOG_LEVEL: info
      REGISTRY_STORAGE_FILESYSTEM_ROOTDIRECTORY: /var/lib/registry
      REGISTRY_AUTH_TOKEN_REALM: https://my-gitlab.com/jwt/auth
      REGISTRY_AUTH_TOKEN_SERVICE: container_registry
      REGISTRY_AUTH_TOKEN_ISSUER: gitlab-issuer
      REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /certs/cert.crt
      REGISTRY_STORAGE_DELETE_ENABLED: "true"
    networks:
      - web
# [...]

Hope this will help.

PS: I obviously made a backup before doing the operation.

[etlam]

Will there be an update of the docs and an “official” way for migrating to the new docker image?

[sachilles]

@etlam Yes, the project-related docs will be updated possible.