Error in Tag Retention Log "Object is not found"

[AlexBarth13]

Expected behavior and actual behavior: I have set up a Tag Retention Policy for housekeeping the images. The Dry Run was working fine and as soon as it was done, I got the information from the Policy log, which images will be kept or deleted. When executing the policy, the images were deleted based on the rule, but in the policy log I see the following error message:

{"errors":[{"code":"NOT_FOUND","message":"{\"code\":10010,\"message\":\"object is not found\",\"details\":\"a83ac447704ac5c98c720eef\"}"}]}

I expected a log, where I get the information about the deleted image.

Steps to reproduce the problem: Step 1: Create a Tag Retention policy with the following configuration.

Step 2: Run the policy

Step 3: After successful run, check the policy log.

Versions: Please specify the versions of following systems.

• harbor version: 2.4

Additional context:

I also checked our log files, if we can see any errors but everything was successful based on the log information.

2022-03-25 08:03:29	
jobservice.log
Mar 25 08:03:29 <host> jobservice[25099]:  Repository: <project>/<image> 

2022-03-25 08:03:31	
jobservice.log
Mar 25 08:03:31 <host> jobservice[25099]: 2022-03-25T07:03:31Z [INFO] [/pkg/retention/job.go:98]: Load 13 candidates from repository <project>/<image>

2022-03-25 08:03:31	
core.log
Mar 25 08:03:31 <host> core[25099]: 2022-03-25T07:03:31Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'RetentionWebhook' on topic 'TAG_RETENTION': TaskID-1325630 Status-SUCCESS Deleted-[<project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[] <project>:<image>:[]] OccurAt-2022-03-25 07:03:31

2022-03-25 08:04:33	
jobservice.log
Mar 25 08:04:33 <host> jobservice[25099]:  Repository: <project>/<image> 

2022-03-25 08:04:35	
jobservice.log
Mar 25 08:04:35 <host> jobservice[25099]: 2022-03-25T07:04:35Z [INFO] [/pkg/retention/job.go:98]: Load 13 candidates from repository <project>/<image>

2022-03-25 08:04:35	
core.log
Mar 25 08:04:35 <host> core[25099]: 2022-03-25T07:04:35Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'RetentionWebhook' on topic 'TAG_RETENTION': TaskID-1325631 Status-SUCCESS Deleted-[<project>:<image>:[]] OccurAt-2022-03-25 07:04:35

2022-03-25 08:04:49	
jobservice.log
Mar 25 08:04:49 <host> jobservice[18398]:  Repository: <project>/<image> 

2022-03-25 08:04:51	
jobservice.log
Mar 25 08:04:51 <host> jobservice[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/retention/job.go:98]: Load 13 candidates from repository <project>/<image>

2022-03-25 08:04:51	
core.log
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'AuditLog' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:51	
core.log
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'Replication' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:51	
core.log
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'ArtifactWebhook' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:51	
core.log 
Mar 25 08:04:51 <host> core[18398]: 2022-03-25T07:04:51Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'DeleteArtifactWebhook' on topic 'DELETE_ARTIFACT': ID-382355, Repository-<project>/<image> Tags-[] Digest-sha256:1d8d3e3bbba68d6b4ec30dced8a9037abd8efdedb46437620b6840809f432dec Operator-harbor-jobservice OccurAt-2022-03-25 07:04:51

2022-03-25 08:04:52	
core.log
Mar 25 08:04:52 <host> core[18398]: 2022-03-25T07:04:52Z [INFO] [/pkg/notifier/notifier.go:205]: Handle notification with Handler 'RetentionWebhook' on topic 'TAG_RETENTION': TaskID-1325632 Status-SUCCESS Deleted-[<project>:<image>:[]] OccurAt-2022-03-25 07:04:52

Are you aware of this error message? If yes, how can we solve it?

Thank you in advance!

Alexander Barth ([email protected]) on behalf of Daimler TSS, Provider Information

[zyyw]

Hi @AlexBarth13, which patch version of harbor 2.4 you were using?

Can't reproduce the error in policy log in harbor 2.4.2 (installed via harbor-helm)? In my case, both dry run and run now reports the log similar to this following one:

2022-03-28T06:45:54Z [INFO] [/pkg/retention/job.go:83]: Run retention process.
 Repository: tce-syu/harbor 
 Rule Algorithm: or 
 Dry Run: false
2022-03-28T06:45:55Z [INFO] [/pkg/retention/job.go:98]: Load 7 candidates from repository tce-syu/harbor
2022-03-28T06:45:55Z [INFO] [/pkg/retention/job.go:201]: 
|                                 Digest                                  |       Tag        | Kind  | Labels |     PushedTime      | PulledTime |     CreatedTime     | Retention |
|-------------------------------------------------------------------------|------------------|-------|--------|---------------------|------------|---------------------|-----------|
| sha256:9ad722a66269e306a27ed110e2b957bf82573125b88a918048c1d40ded7e6fba | 2.2.3,9ad722a662 | image |        | 2022/03/28 06:40:02 |            | 2022/03/28 06:40:02 | RETAIN    |
| sha256:d32621605c1bf43aabd4a578e397029a5774a475a4d1620a630f1bcb4ec63513 | d32621605c       | image |        | 2022/03/28 06:40:01 |            | 2022/03/28 06:40:01 | RETAIN    |
| sha256:9f1ca138dc2e59e2e9fd9521f32b98e1cc2c8ad8837f12bb5d3bc3da2e1c49f8 | 9f1ca138dc       | image |        | 2022/03/28 06:40:00 |            | 2022/03/28 06:40:00 | RETAIN    |
| sha256:7a12dc4d2e051519ff5d7ba1f13993fd340de20adfa9fe272e9245be62802795 | 7a12dc4d2e       | image |        | 2022/03/28 06:39:59 |            | 2022/03/28 06:39:59 | RETAIN    |
| sha256:610c0ec78afd7095eed05624976ab83bd05c41b1f926fb2ccb7cf1bdfc239f37 | 2.3.3,610c0ec78a | image |        | 2022/03/28 06:39:58 |            | 2022/03/28 06:39:57 | DEL       |
| sha256:45ee679e2a57d97aaf2228e229901752f961b2a82f1e691344406be92aaf1a7a | 2.4.0            | image |        | 2022/03/28 06:39:56 |            | 2022/03/28 06:39:56 | DEL       |
| sha256:c4e9d26468417f0fbce79e0b807ce1aad8e5e0cab26a018e1de1bed3630c497b | 2.4.2            | image |        | 2022/03/28 06:39:55 |            | 2022/03/28 06:39:55 | DEL       |

[AlexBarth13]

Hi @zyyw Thanks for your reply! We are using Harbor v2.4 currently.

Alexander Barth ([email protected]) on behalf of Daimler TSS, Provider Information

[zyyw]

Hi @AlexBarth13, do you have the timestamp of the error message from policy log?

The Policy log only kept for 1 day. So the those policy log exceeds 1 day may not exist.

[AlexBarth13]

Hi @zyyw Thanks for the hint with the policy log. Since the policy log from my inital test was already removed, I have done a new test with the same outcame. But in the policy log I can see the log:

2022-03-28T08:56:50Z [INFO] [/pkg/retention/job.go:83]: Run retention process.
 Repository: <project>/<artifact>
 Rule Algorithm: or
 Dry Run: false
2022-03-28T08:56:52Z [INFO] [/pkg/retention/job.go:98]: Load 12 candidates from repository <project>/<artifact>
2022-03-28T08:56:52Z [INFO] [/pkg/retention/job.go:201]:
|                                 Digest                                  |  Tag   | Kind  | Labels |     PushedTime      |     PulledTime      |     CreatedTime     | Retention |
|-------------------------------------------------------------------------|--------|-------|--------|---------------------|---------------------|---------------------|-----------|
| sha256:6f7a3daa1b8dcd12d4eca0ed267ec3895066ca3730b8e6542430e3436a193b16 | latest | image |        | 2021/11/03 11:21:06 | 2022/02/04 17:55:54 | 2021/11/03 11:21:06 | RETAIN    |
| sha256:fca4321bf50e7085a705f70a46849a5842b438dfa5eef8fd2a74b33724b66ff4 |        | image |        | 2021/11/03 11:05:01 | 2022/01/18 09:55:14 | 2021/11/03 11:05:01 | RETAIN    |
| sha256:d5ac92ff2613c9b64b2f52b2c3cd2e7b931b216644b051d330c121772207ed77 |        | image |        | 2021/11/03 10:54:03 | 2022/01/18 09:58:00 | 2021/11/03 10:54:03 | RETAIN    |
| sha256:ccf5a8f4eb6a9e833307da4e42bd745dce35d425e3ec80de3215098889441c0f |        | image |        | 2021/11/03 10:46:58 | 2022/02/21 02:10:33 | 2021/11/03 10:46:58 | RETAIN    |
| sha256:eb00052912b0a143f188de8b333a081a3f1155c9ef628c366579505fea9d1edd |        | image |        | 2021/11/02 16:20:59 | 2022/01/26 12:41:06 | 2021/11/02 16:20:59 | RETAIN    |
| sha256:7ec6ac7bed619b39d7864483038f0b8915b14e54af824a5a6d806c6957ea8079 |        | image |        | 2021/11/02 16:09:44 | 2021/11/02 16:09:58 | 2021/11/02 16:09:44 | RETAIN    |
| sha256:7f4cc6c39b238d7575131db31eb373ae60ee81cd8f230368e3585b11517d4f25 |        | image |        | 2021/11/02 16:00:07 | 2021/11/02 16:03:32 | 2021/11/02 16:00:07 | RETAIN    |
| sha256:6e48c3f667f16f4fd4cbdea95c203c7f887ebad9ab82566cd2797ca93b966c3c |        | image |        | 2021/10/29 12:42:21 | 2021/11/02 15:46:41 | 2021/10/29 12:42:21 | RETAIN    |
| sha256:742f7a768762cd54116ac351e327a7958e1868a817dd735325564b27662fa003 |        | image |        | 2021/10/29 12:35:30 | 2021/10/29 12:35:48 | 2021/10/29 12:35:30 | RETAIN    |
| sha256:dbc32ea725f900209e97979d608971dadc56284411659860859e55d94e909106 |        | image |        | 2021/10/29 11:48:10 | 2021/10/29 11:49:56 | 2021/10/29 11:48:10 | RETAIN    |
| sha256:7ccc1e154de101a80dae2ed14958189fde0adc358cfc71370f7fe207be3626fd |        | image |        | 2021/10/29 11:23:45 | 2021/10/29 11:37:55 | 2021/10/29 11:23:45 | RETAIN    |
| sha256:e7c6348167bfbf14c93b28815b327d96cf519ffe57ed12d5dd9008aa53f7b0de |        | image |        | 2021/10/29 10:25:06 | 2021/10/29 11:22:20 | 2021/10/29 10:25:06 | DEL       |

Alexander Barth ([email protected]) on behalf of Daimler TSS, Provider Information

[zyyw]

@AlexBarth13 since you can see the policy log with the new test as expected, shall we close the issue?

[AlexBarth13]

Hi @zyyw! No. I can see the log file itself on our server, but from the Harbor UI I still see the error message object is not found. Our customers of our platform only have access to the Harbor UI and they should see this there.

[zyyw]

@AlexBarth13 do you mind sharing a screenshot of the error message object is not found?

[AlexBarth13]

@zyyw Sure!

Step 1: Open the run and click on the Log icon.

Step 2: Error message is directly displayed

[zyyw]

I just tried with harbor 2.4.2 with a similar tag retention rule in the description of this issue. And clicked DRY RUN first and then clicked RUN NOW. In both case, click the Log icon as you red rectangle-d above. And then see output in a separate browser window, like this: https://github.com/goharbor/harbor/issues/16592#issuecomment-1080264424 No error message like step2 you mentioned above displays.

[AlexBarth13]

@zyyw We are using the version 2.4.0.

I just did another test like you do - first with ´DRY RUN´ and than ´RUN´.

Output of ´DRY RUN´:

Output of ´RUN´:

[zyyw]

@AlexBarth13 Odd! Just set-up a Harbor 2.4.0 instance, configured a retention rule similar to the one in the description of this issue. First DRY RUN, then RUN NOW, in both case, click the Log icon, output in a separate browser like this:

2022-03-29T09:18:47Z [INFO] [/pkg/retention/job.go:83]: Run retention process.
 Repository: tce-syu/harbor 
 Rule Algorithm: or 
 Dry Run: false
2022-03-29T09:18:47Z [INFO] [/pkg/retention/job.go:98]: Load 7 candidates from repository tce-syu/harbor
2022-03-29T09:18:47Z [INFO] [/pkg/retention/job.go:201]: 
|                                 Digest                                  |       Tag        | Kind  | Labels |     PushedTime      | PulledTime |     CreatedTime     | Retention |
|-------------------------------------------------------------------------|------------------|-------|--------|---------------------|------------|---------------------|-----------|
| sha256:9ad722a66269e306a27ed110e2b957bf82573125b88a918048c1d40ded7e6fba | 2.2.3,9ad722a662 | image |        | 2022/03/29 09:16:31 |            | 2022/03/29 09:16:31 | RETAIN    |
| sha256:d32621605c1bf43aabd4a578e397029a5774a475a4d1620a630f1bcb4ec63513 | d32621605c       | image |        | 2022/03/29 09:16:30 |            | 2022/03/29 09:16:30 | RETAIN    |
| sha256:9f1ca138dc2e59e2e9fd9521f32b98e1cc2c8ad8837f12bb5d3bc3da2e1c49f8 | 9f1ca138dc       | image |        | 2022/03/29 09:16:30 |            | 2022/03/29 09:16:30 | RETAIN    |
| sha256:7a12dc4d2e051519ff5d7ba1f13993fd340de20adfa9fe272e9245be62802795 | 7a12dc4d2e       | image |        | 2022/03/29 09:16:30 |            | 2022/03/29 09:16:30 | RETAIN    |
| sha256:610c0ec78afd7095eed05624976ab83bd05c41b1f926fb2ccb7cf1bdfc239f37 | 2.3.3,610c0ec78a | image |        | 2022/03/29 09:16:29 |            | 2022/03/29 09:16:29 | RETAIN    |
| sha256:45ee679e2a57d97aaf2228e229901752f961b2a82f1e691344406be92aaf1a7a | 2.4.0            | image |        | 2022/03/29 09:16:29 |            | 2022/03/29 09:16:29 | DEL       |
| sha256:c4e9d26468417f0fbce79e0b807ce1aad8e5e0cab26a018e1de1bed3630c497b | 2.4.2            | image |        | 2022/03/29 09:16:28 |            | 2022/03/29 09:16:28 | DEL       |

No error message displayed.

[AlexBarth13]

Hi @zyyw! I might have some idea why I will get the error message. Let me take a deeper look into it. I will get back to you.

[AlexBarth13]

Hi @zyyw On Tuesday we had a call with @Vad1mo and @OrlinVasilev about general stuff. Also, we talked about this issue. Vadim confirmed my guess that the issue is related to our infrastructure. We are using multiple VMs with an load balancer in front of it. This means my request will be routed randomly between the VMs. If I am on a VM, where the job log is not available, than I can see the above mentioned issue.

We are currently checking, how we can change the way where the job logs are safed.

Alexander Barth ([email protected]) on behalf of Mercedes-Benz Tech Innovation GmbH, Provider Information

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[AlexBarth13]

This issue can be closed because we found a solution to fix the above mentioned error message. Our solution is, that we created an NFS share to store the job logs on this share instead on a local VM.

Alexander Barth ([email protected]) on behalf of Mercedes-Benz Tech Innovation GmbH, Provider Information

[Jeansen]

I have a similar issue. Entries older than 2 days give me the aforementioned error. The other two show log content, as expected. Either I am doing something wrong or have missed a configuration or something else is wrong. In my case, harbor runs on a separate system, singel instance. But I think the UI should only show, what is available. If the log is not available (e.g. because of log rotation), then it should also not be present in the UI. There's not much value int it anyway, if one cannot access the logs.

[bonniechen1119]

I also meet this issue,please help me how to fix the above mentioned error message

[Egreg0r]

I also meet this issue, on 2.6.2 version. Upd. In my case, helped changing the owner to 10000 of the external (in host) directory for jobservice and redis.

[adriannieto-attechnest]

This is still an issue, please re-oppen

[OrlinVasilev]

@adriannieto-attechnest can you share some more information about the issue you are facing ?

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[smonko]

We had same issue. Logs from jobs older than 2 days was not available.

{"errors":[{"code":"NOT_FOUND","message":"{\"code\":10010,\"message\":\"object is not found\",\"details\":\"6731b93bdec50c0560db4fa7\"}"}]}

Solution looks like is to change logger_sweeper_duration setting for jobservice:

this is original setting from harbor.yaml

jobservice:
  max_job_workers: 10
  job_loggers:
    - STD_OUTPUT
    - FILE
  logger_sweeper_duration: 1 #day  <--- Increase this value

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.