harbor icon indicating copy to clipboard operation
harbor copied to clipboard

Published 20 hours ago • verified publisher icon goharbor

GC is failing

Open AyushCloud opened this issue 3 years ago • 9 comments

Hello,

I am getting error in GC Digest-sha256:7bfc9d510740b604ebe7d5d3069508d00fa46515f6ac90bfbf914c606fdcd693 CreationTime-2020-12-11 15:28:50 2021-05-14T00:00:33Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:201]: no need to execute GC as there is no non referenced artifacts. 2021-05-14T00:00:33Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:321]: The GC job actual frees up 0 MB space. 2021-05-14T00:00:33Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:351]: failed to clean registry cache error retrieving 'blobs::' keys, pattern blobs::

Harbor Version v2.2.0 Docker version 19.0.0 docker-compose version 1.26

AyushCloud avatar May 19 '21 11:05 AyushCloud

could you verify the network about harbor core/jobservice and redis? Are there more details about the failure in the jobservcie log?

wy65701436 avatar May 21 '21 08:05 wy65701436

Hello,

Yes, there in no issue in the network connectivity Currently i could see only this error in the jobservice logs.

Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:273]: delete manifest from storage: sha256:3405303866196bcdbf473e3aa2caf1915263e2aa6de0479075b1f4253367ac59 Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:289]: delete artifact trash record from database: 22409, devenvi206/ds-odc-health-exporter, sha256:3405303866196bcdbf473e3aa2caf1915263e2aa6de0479075b1f4253367ac59 Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:301]: delete blob from storage: sha256:3405303866196bcdbf473e3aa2caf1915263e2aa6de0479075b1f4253367ac59 Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:319]: delete blob record from database: 87213, sha256:3405303866196bcdbf473e3aa2caf1915263e2aa6de0479075b1f4253367ac59 Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:301]: delete blob from storage: sha256:dd5c701912f8cbda15d5d10b66fe3f4a9589a3e754bdda1de97315485086b6fe Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:319]: delete blob record from database: 87212, sha256:dd5c701912f8cbda15d5d10b66fe3f4a9589a3e754bdda1de97315485086b6fe Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:331]: The GC job actual frees up 1276 MB space. Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:361]: failed to clean registry cache error retrieving 'blobs::' keys, pattern blobs:: Dec 5 00:03:00 jobservice[13934]: 2021-12-05T00:03:00Z [ERROR] [/jobservice/runner/redis.go:113]: Job 'GARBAGE_COLLECTION:4ed0b8dc7f4d4583991343d5' exit with error: run error: error retrieving 'blobs::*' keys

AyushCloud avatar Dec 06 '21 06:12 AyushCloud

The keys strored in Redis are like this: 127.0.0.1:6379> keys *

  1. "5f9133a01e1c53fec97bebc930278b6d"
  2. "f56b4a352f38a538abfb1d78c7917cb3"
  3. "d393b40291ad9fa382c26a55114dce27"
  4. "7c63208b20d9e27c01d9f77e480d1037"
  5. "9bee43c50d30bc479cc05abc57253079"
  6. "1ba2517bd2658d1ad6f9095b419b352d"
  7. "64620be56de0183308d8dded68637625"
  8. "b8f8f11f42d759bbb1b3e4bcac489ffc"
  9. "69857e3a239aceb2f79d6011dcd54ef9"
  10. "a3a5fe1ed59b3488a1398e1a19cc5c5c"
  11. "bccf30c7368a1882a5fc26867801355b"
  12. "70f1529826ba49015bd361bc7b7482d4"
  13. "87cf04b95e11a0fa6cd1df69f7291895"
  14. "70ba5dc96cb38192c4ebef2233bd4f7d"
  15. "ade43299879f2bf73fa8b49df289c45c"
  16. "e9e78bbbd8037de8bb4f841c04e88cca"
  17. "ac6db0dabc4503832ebaf75672a774cb"
  18. "8ccba3df6b54035d09ea7e152a5bdac4"
  19. "65e64dff4df4cdd244d3e2d7699d06eb"

But i guess it should be like

// sample of keys in registry redis: // 1) "blobs::sha256:1a6fd470b9ce10849be79e99529a88371dff60c60aab424c077007f6979b4812" // 2) "repository::library/hello-world::blobs::sha256:4ab4c602aa5eed5528a6620ff18a1dc4faef0e1ab3a5eddeddb410714478c67f"

Am I missing something?

AyushCloud avatar Dec 07 '21 10:12 AyushCloud

Hey 👋 are there any updates to this issue? I'm experiencing the same issue using a external redis and database for harbor.

niklasweimann avatar Dec 09 '21 13:12 niklasweimann

Any update on the issue ?

AyushCloud avatar Feb 22 '22 08:02 AyushCloud

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Jul 06 '22 09:07 github-actions[bot]

Any update on this issue ?

AyushCloud avatar Jul 06 '22 09:07 AyushCloud

Having similar issue here:

2022-07-20T07:06:07Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:317]: [3643/3643] delete blob from storage: sha256:68d445d604042f41c8cae4c928ffa1b74554d580e6039025655353fcd3e04f31
2022-07-20T07:06:07Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:337]: [3643/3643] delete blob record from database: 1234306, sha256:68d445d604042f41c8cae4c928ffa1b74554d580e6039025655353fcd3e04f31
2022-07-20T07:06:07Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:356]: 2901 blobs and 733 manifests are actually deleted
2022-07-20T07:06:07Z [INFO] [/jobservice/job/impl/gc/garbage_collection.go:357]: The GC job actual frees up 59701 MB space.
2022-07-20T07:06:07Z [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:387]: failed to clean registry cache error retrieving 'blobs::*' keys, pattern blobs::*

Antiarchitect avatar Jul 20 '22 17:07 Antiarchitect

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

github-actions[bot] avatar Sep 19 '22 09:09 github-actions[bot]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.

github-actions[bot] avatar Oct 19 '22 09:10 github-actions[bot]

If you came here looking for a fix:

In our case the problem was that we updated the domain to Redis recently and the scheduled Garbage Collection did not work until then. Two issues cause this:

  1. Harbor persists the Redis Endpoint to DB (#17984)
  2. Jobservice swallows the error message from the redis library, giving a misleading error message (#17986)

TL;DR: If you get this error after updating the redis endpoint, change your GC Schedule and it should work again.

czenker avatar Dec 13 '22 10:12 czenker

I have the same issue with harbor v2.8.2 [ERROR] [/jobservice/job/impl/gc/garbage_collection.go:434]: failed to clean registry cache error retrieving 'blobs::*' keys: dial tcp: lookup redis on 127.0.0.11:53: no such host, pattern blobs::*

Used external Redis Sentinel instance. I've changed GC Shedule but this does not solved error. Is there any solution to this problem?

pavel-z1 avatar Jul 06 '23 07:07 pavel-z1