Garbage Collection in Harbor Not Freeing Up Expected Storage Space While the UI Shows Space Has Been Freed Up

[ElieAzar-Algo]

Expected behavior and actual behavior:

I encountered a problem with the cleanup of unused Docker images due to full storage on my server.

Expected Behavior:

The mounted storage pointing to the Docker Images directory is full:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb 197G 197G 0G 100% /Docker_Images

To free up space, I applied the following retention policy to the project with approximately 125GB of images:

• For repositories matching **, retain the most recently pushed 5 artifacts with tags matching ** and include untagged artifacts.

After running the retention policy, I observed the following result in the UI:

Next, I went to the "Clean Up" section, ran the Garbage Collection, and received this output:

• Mode: Manual
• Dry Run: No
• Result: SUCCESS
• Details: 15710 blob(s) and 5974 manifest(s) deleted, 120.57GiB space freed up
• Time: Aug 28, 2024, 7:05:44 PM - Aug 28, 2024, 7:10:01 PM
  

The expected result was for the images to be cleaned up and the storage to reflect something around the following:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb 197G 77G 120G 39% /Docker_Images

Actual Behavior:

However, the actual result is that the Garbage Collection only freed up about 10GB of space, despite the UI indicating that 15710 blob(s) and 5974 manifest(s) were deleted, freeing up 120.57GiB, as shown in the screenshot above.

Steps to reproduce the problem:

1. Confirm that the storage mounted on the server for Docker images is full:

• Example:

Filesystem Size Used Avail Use% Mounted on
/dev/sdb 197G 197G 0G 100% /Docker_Images

2. Go to the project in Harbor with the highest size of Docker images.

3. Apply the following retention policy:

• For repositories matching **, retain the most recently pushed 5 artifacts with tags matching **, and include untagged artifacts.

4. Run the retention policy and observe the result in the UI. as shown above in the screenshot that everything is going as expected

5. Go to the "Clean Up" section and run the Garbage Collection with the following settings:

• Mode: Manual
• Dry Run: No

6. Note the output that shows the following:

• Result: SUCCESS
• Details: 15710 blob(s) and 5974 manifest(s) deleted, (ex: 120.00GiB) space freed up as shown above in the screenshot

7. Check the server storage again using the df -h command:

Versions:

Please specify the versions of following systems.

• harbor version: 2.10.0
• docker engine version: 25.0.2
• docker-compose version: 1.27.4

Additional context:

Harbor config files:

• Find the config file attached as txt file because github doesn't support yaml files in the attch feature. harbor-yml.txt

[wy65701436]

can you try to restart harbor? and then check the storage again.

[ElieAzar-Algo]

Thank you, @wy65701436, for your response.

I restarted Harbor, but the storage issue persists. In fact, we've done multiple restarts over the past few days without any improvement.

[linuxmail]

hi,

see the exact same issue. Installed today with online installer v1.10.19. I've also tried to delete the whole project(s) but the disk usage is the same.

Update:

Maybe because of:

To avoid damaging the uploading artifact, the garbage collection introduces a time windows(2 hours) to reserve the recent uploaded layers.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[wy65701436]

@ElieAzar-Algo Are you using the local file system? Also, could you retrieve one or more deleted digests from the GC log and check whether they exist on your disk?

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.