Garbage collection failing causing registry to fail

[ppapp92]

I'm currently running kube-image-keeper with the built-in MinIO implementation. Everything works fine until the weekly garbage collection cronjob runs. The job reports that the Access Key Id provided is not found and fails.

deployment "kube-image-keeper-registry" successfully rolled out
failed to garbage collect: failed to mark: s3aws: InvalidAccessKeyId: The Access Key Id you provided does not exist in our records.
    status code: 403, request id: 17AE59CBDF8076F4, host id: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855command terminated with exit code 1
Stream closed EOF for kube-image-keeper/kube-image-keeper-registry-garbage-collection-28440000-mg2jx (kubectl)

However after the cronjob runs the kube-image-keeper-registry also starts to fail with 503 errors not being able to connect to MinIO. The only solution is to have the MinIO provisioner job to rerun and then kube-image-keeper-registry is able to reconnect.

Here is a sample override of the Helm Chart

kube-image-keeper:
  minio:
    enabled: true
    auth:
      existingSecret: minio-root-auth
    persistence:
      storageClass: ebs-sc
      size: 100Gi

kind: Secret
apiVersion: v1
metadata:
  name: minio-root-auth
type: Generic
stringData:
  root-user: <ROOT_USER>
  root-password: <ROOT_PASSWORD>

Using version 1.5.0

[Nicolasgouze]

Hi @ppapp92 , we have the exact same config running on our internal env & do not reproduce this issue (using for exammple 1.7.0 beta 2). Do you still have the issue ? Can you still give a try with the 1.7.0 beta 2 ?

[Nicolasgouze]

Closing this ticket. @ppapp92, feel free to come back to us if you still reproduce with a recent release. (1.7 or 1.8)