Garbage collection running wild

[danielbecker]

Description

Since old tags for our images were piling up, I enabled garbage collection to run with keep_latest: 10 and older_than: 90. From the documentation, I assume this means remove all tags but the last 10 and don't consider any tag younger than 90 days.

However, what happened was a ton of old, admin and template images being removed entirely. The images weren't updated recently, but frequently pulled, as we need them in our daily deployment process.

Steps to reproduce

1. Enable garbage collection:

• PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
• PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=90
• PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=10

2. Restart portus docker-compose up -d
3. Watch your images diappear

• Expected behavior: Remove tags older than 90 days and keep at least 10 around
• Actual behavior: Entire images were removed

Providing logs of the moment when the issue has happened would also be useful. If you are in production, you might want to set the PORTUS_LOG_LEVEL to debug to get a more verbose log.

Deployment information

Deployment method: docker-compose.yml

version: "3.7"

services: portus: image: opensuse/portus:2.4 restart: unless-stopped environment: - PORTUS_MACHINE_FQDN_VALUE=xxx

  # DB. The password for the database should definitely not be here. You are
  # probably better off with Docker Swarm secrets.
  - PORTUS_DB_HOST=db
  - PORTUS_DB_DATABASE=xxx
  - PORTUS_DB_PASSWORD=xxx
  - PORTUS_DB_POOL=5

  # Secrets. It can possibly be handled better with Swarm's secrets.
  - PORTUS_SECRET_KEY_BASE=xxx
  - PORTUS_KEY_PATH=/certificates/2019_key.pem
  - PORTUS_PASSWORD=xxx

  # SSL
  - PORTUS_PUMA_TLS_KEY=/certificates/2019_key.pem
  - PORTUS_PUMA_TLS_CERT=/certificates/2019_cert.pem

  - RAILS_SERVE_STATIC_FILES=false
ports:
  - 127.0.0.1:8063:3000
links:
  - db
volumes:
  - /srv/xxx/portus/certs:/certificates:ro
  - /srv/xxx/portus/assets:/srv/Portus/public
extra_hosts:
  - "xxx:192.168.252.113"
logging:
  driver: "gelf"
  options:
    gelf-address: "tcp://xxx:5555"
    tag: "portus-frontend"
    mode: non-blocking
    max-buffer-size: 1g

background: image: opensuse/portus:2.4 restart: unless-stopped environment: - CCONFIG_PREFIX=PORTUS - PORTUS_MACHINE_FQDN_VALUE=xxx

  # DB. The password for the database should definitely not be here. You are
  # probably better off with Docker Swarm secrets.
  - PORTUS_DB_HOST=db
  - PORTUS_DB_DATABASE=xxx
  - PORTUS_DB_PASSWORD=xxx
  - PORTUS_DB_POOL=5

  # Secrets. It can possibly be handled better with Swarm's secrets.
  - PORTUS_SECRET_KEY_BASE=xxx
  - PORTUS_KEY_PATH=/certificates/2019_key.pem
  - PORTUS_PASSWORD=xxx

  # Enable garbage collection
  - PORTUS_DELETE_GARBAGE_COLLECTOR_ENABLED=true
  - PORTUS_DELETE_GARBAGE_COLLECTOR_OLDER_THAN=90
  - PORTUS_DELETE_GARBAGE_COLLECTOR_KEEP_LATEST=10

  - PORTUS_BACKGROUND=true
links:
  - db
volumes:
  - /srv/xxx/portus/certs:/certificates:ro
extra_hosts:
  - "xxx:192.168.252.113"
logging:
  driver: "gelf"
  options:
    gelf-address: "tcp://xxx:5555"
    tag: "portus-backend"
    mode: non-blocking
    max-buffer-size: 1g

db: image: library/mariadb:10.0.23 restart: unless-stopped command: mysqld --character-set-server=utf8 --collation-server=utf8_unicode_ci --init-connect='SET NAMES UTF8;' --innodb-flush-log-at-trx-commit=0 environment: - MYSQL_DATABASE=xxx - MYSQL_ROOT_PASSWORD=xxx volumes: - /srv/xxx/portus/database:/var/lib/mysql

registry: image: library/registry:2.6 restart: unless-stopped command: ["/bin/sh", "/etc/docker/registry/init"] environment: # Authentication REGISTRY_AUTH_TOKEN_REALM: https://xxx/v2/token REGISTRY_AUTH_TOKEN_SERVICE: xxx REGISTRY_AUTH_TOKEN_ISSUER: xxx REGISTRY_AUTH_TOKEN_ROOTCERTBUNDLE: /secrets/2019_bundle.pem

  # SSL
  REGISTRY_HTTP_TLS_CERTIFICATE: /secrets/2019_cert.pem
  REGISTRY_HTTP_TLS_KEY: /secrets/2019_key.pem

  # Portus endpoint
  REGISTRY_NOTIFICATIONS_ENDPOINTS: >
    - name: portus
      url: https://xxx/v2/webhooks/events
      timeout: 2000ms
      threshold: 5
      backoff: 1s
volumes:
  - /srv/xxx/portus/registry/storage:/var/lib/registry
  - /srv/xxx/portus/certs:/secrets:ro
  - /srv/xxx/portus/registry/config/config.yml:/etc/docker/registry/config.yml:ro
  - /srv/xxx/portus/registry/init/init:/etc/docker/registry/init:ro
ports:
  - 127.0.0.1:xxx:5000
  - 127.0.0.1:xxx:5001 # required to access debug service
links:
  - portus:portus
logging:
  driver: "gelf"
  options:
    gelf-address: "tcp://xxx:5555"
    tag: "portus-registry"
    mode: non-blocking
    max-buffer-size: 1g

Configuration:

Please note that I had to turn off garbage collection for now to stop portus from causing more damage.

[Mailer config] Host: portus.test.lan [Mailer config] Protocol: https:// Evaluated configuration:

email: from: [email protected] name: Portus reply_to: '' smtp: enabled: false address: smtp.example.com port: 587 domain: example.com ssl_tls: '' enable_starttls_auto: false openssl_verify_mode: none ca_path: '' ca_file: '' user_name: '' password: "" authentication: login gravatar: enabled: true delete: enabled: true contributors: false garbage_collector: enabled: false older_than: 30 tag: '' ldap: enabled: false hostname: ldap_hostname port: 389 timeout: 5 encryption: method: '' options: ca_file: '' ssl_version: TLSv1_2 base: '' filter: '' uid: uid authentication: enabled: false bind_dn: '' password: "" guess_email: enabled: false attr: '' oauth: local_login: enabled: true google_oauth2: enabled: false id: '' secret: '' domain: '' options: hd: '' open_id: enabled: false identifier: '' domain: '' github: enabled: false client_id: '' client_secret: '' organization: '' team: '' domain: '' gitlab: enabled: false application_id: '' secret: '' group: '' domain: '' server: '' bitbucket: enabled: false key: '' secret: '' domain: '' options: team: '' first_user_admin: enabled: true signup: enabled: true check_ssl_usage: enabled: true registry: jwt_expiration_time: value: 15 catalog_page: value: 100 timeout: value: 2 read_timeout: value: 120 machine_fqdn: value: portus.easybell.de display_name: enabled: false user_permission: change_visibility: enabled: true create_team: enabled: true manage_team: enabled: true create_namespace: enabled: true manage_namespace: enabled: true create_webhook: enabled: true manage_webhook: enabled: true push_images: policy: allow-teams security: clair: server: '' health_port: 6061 timeout: 900 zypper: server: '' dummy: server: '' anonymous_browsing: enabled: true background: registry: enabled: true sync: enabled: true strategy: initial pagination: limit: 10 per_page: 10 before_after: 2

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952

version: 0.1
storage:
  filesystem:
    rootdirectory: /var/lib/registry
  delete:
    enabled: true
http:
  addr: 0.0.0.0:5000
  debug:
    addr: 0.0.0.0:5001

Portus version: 2.4.3@5a616c0ef860567df5700708256f42505cdb9952

[Bonko]

As far as i know this feature was planned for Portus 2.5: https://github.com/SUSE/Portus/issues/2051

Sadly the documentation refers to master instead of current stable

[stale[bot]]

Thanks for all your contributions! This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.