Harbor does not clean up chart entry in index.yaml when chart is deleted

[agill17]

If you are reporting a problem, please make sure the following information are provided:

Expected behavior and actual behavior: The chart entry for the specific version should no longer exist in index.yaml

Steps to reproduce the problem: Upload multiple versions of any chart Delete 1 version of that chart Look at index.yaml and it will still have the deleted version in index.yaml file

Versions: Please specify the versions of following systems.

• harbor version: 2.1.1
• docker engine version: 19.0.3
• docker-compose version: [z.z.z]

Additional context:

• Harbor is hosted on a k8s cluster, installed using helm chart
• Storage backend is s3
• Using chartmuseum
• Redis is hosted on Elasticache.

[stonezdj]

@agill17 Is your s3 storage a customer s3?

[agill17]

@agill17 Is your s3 storage a customer s3?

@stonezdj I dont understand what "customer s3" means? I have a s3 bucket where all my artifacts are being stored for both docker images and helm charts. Thats where the index-cache.yaml also lives which does not clean up chart version entry once a chart is deleted from harbor

[Fgruntjes]

Same issue here, checked the S3 bucket and when I delete the index-cache.yaml manually its just regenerated with the missing charts.

[mohag]

@Fgruntjes What works for me (running on Kubernetes):

• Scale down Redis (mine is using emptyDir stoage) and chartmuseum to 0.
• Delete the index-cache.yaml
• Scale chartmuseum and redis back up.

Partially based on this comment

[Fgruntjes]

@Fgruntjes What works for me (running on Kubernetes):

• Scale down Redis (mine is using emptyDir stoage) and chartmuseum to 0.
• Delete the index-cache.yaml
• Scale chartmuseum and redis back up.

Partially based on this comment

Thank you for your message, greatly appreciated. We also managed with that same comment. Sorry, I forgot to post a reply here.

Still a bug, but we do have a workaround.

[RichardMills]

@Fgruntjes What works for me (running on Kubernetes):

• Scale down Redis (mine is using emptyDir stoage) and chartmuseum to 0.
• Delete the index-cache.yaml
• Scale chartmuseum and redis back up.

Partially based on this comment

We also had success doing this, with a few caveats. We started with a chartmuseum that /thought/ it had 40k+ charts with a 65MB index-cache.yaml file in S3. In reality, we only had about 5000 charts in S3 after running a bunch of cleanup scripts through the harbor API.

First, I'm not sure if you actually need to delete the index-cache.yaml. We tried a few different times and even after deleting and even putting back the 65MB cache file in S3. In the end, even with the 65MB cache file there, rebooting our elasticache redis server in AWS caused it to rebuild the cache and store a more reasonable 9MB cache file. This reduced our 30-40s time for helm repo add of that repository to a tolerable 5s.

Second, (our biggest trouble) we learned that chartmuseum causes a large CPU spike while it restarts and rebuilds the cache after rebooting redis. Ours (deployed in kubernetes) caused a 3 CPU spike that overwhelmed our 2CPU nodes that were running harbor and chartmuseum pods. It actually crashed the nodes and caused them to become unresponsive. We had to switch from m5.large to m5.xlarge nodes (4 CPU) and increase resource limit to 3000m (3 CPU) for chartmuseum to properly boot.

[ninjadq]

the index-cache.yaml file and the cache entry in redis should be deleted at the same time. Or it will regenerate each other

[ninjadq]

if you have too many charts in one project, it will cause the big size index-cache.yaml issue. The same as CPU spike when chartmuseum starting. their mechanism is by design in chartmuseum.

If possible, you can try the helm OCI experience feature.

[muhittink]

Any updates here? We are facing the same Problem with Harbor 2.4 and the Chartmuseum. Deleting the index chache manually will regenerate the index cache and everything is fine. We are trying to push our signed charts over OCI with helm v3.8 but it does not upload the .prov files as described in the manual.

[muhittink]

Our workaround was to disable the index cache. I think this can lead to performance Issues with larger instances.

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[1-cameron]

Also having this issue in v2.5.3 with Chartmuseum. Still need to delete the index cache files and redis entries when deleting some versions of charts.

[mohag]

versions of charts.

The main issue I had - charts deleted from the UI (and likely API) still listed there seem gone when using Chartmuseum >= 0.13.1. (Deleting them directly from the storage still needs tricks, but that is likely an unsupported method of deleting them) (I'm using a chart from a third-party where the chartmuseum version might differ though)

How are you deleting the chart versions?

[github-actions[bot]]

This issue is being marked stale due to a period of inactivity. If this issue is still relevant, please comment or remove the stale label. Otherwise, this issue will close in 30 days.

[github-actions[bot]]

This issue was closed because it has been stalled for 30 days with no activity. If this issue is still relevant, please re-open a new issue.