helm
[helm-3.18.0] Duplicated upload when using helm cli to upload an image.
when using helmOCI client to upload package with tag, its uploading it twice - once with the tag and another time with the SHA
scenario to reproduce: <in my case uploading to Artifactory>
- helm registry login -u myUser -p myPassword myartifactory.com
- helm create busybox
- helm package busybox --version 100.100.100
- helm push busybox-100.100.100.tgz oci://myartifactory.com/sobhe
PUT /api/docker/sobhe/v2/busybox/manifests/sha256:065fa59542e3d2ce4006241e41069e387eb0958235cdec165f9ceaf334228ffd -> 201 PUT /api/docker/sobhe/v2/busybox/manifests/100.100.100 -> 201 ----the main point here is: "you guys push the same manifest twice, once by tag and once by digest"
all of this is not happening on the previous versions like 3.17.3
Using debug code from https://github.com/helm/helm/pull/30917)...
Here are the transactions to push a brand new chart to a non existent repository:
% helm push --debug --plain-http busybox-100.100.101.tgz oci://127.0.0.1:15000/fdsa
level=DEBUG msg=Request id=0 url=http://127.0.0.1:15000/v2/aaaa/busybox/manifests/sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d method=HEAD header=" \"Accept\": \"application/vnd.docker.distribution.manifest.v2+json, application/vnd.docker.distribution.manifest.list.v2+json, application/vnd.oci.image.manifest.v1+json, application/vnd.oci.image.index.v1+json, application/vnd.oci.artifact.manifest.v1+json\"\n \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=0 status="404 Not Found" header=" \"Content-Length\": \"281\"\n \"Access-Control-Allow-Headers\": \"Authorization,content-type,X-ZOT-API-CLIENT\"\n \"Access-Control-Allow-Methods\": \"HEAD,GET,DELETE,OPTIONS\"\n \"Access-Control-Allow-Origin\": \"*\"\n \"Content-Type\": \"application/json\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"" body=" No response body to print"
level=DEBUG msg=Request id=1 url=http://127.0.0.1:15000/v2/aaaa/busybox/blobs/sha256:d7d6aa55bf3d53af143d0a454758d8a7fa4ce4f5e5c4a49c7140e88529e5b6a2 method=HEAD header=" \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Request id=2 url=http://127.0.0.1:15000/v2/aaaa/busybox/blobs/sha256:f8c71ec7aa5e0cca02e33bbd7678b947bac803b893cb0015cb4707f01fa3721e method=HEAD header=" \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=1 status="404 Not Found" header=" \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"405\"\n \"Content-Type\": \"application/json\"" body=" No response body to print"
level=DEBUG msg=Request id=3 url=http://127.0.0.1:15000/v2/aaaa/busybox/blobs/uploads/ method=POST header=" \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=2 status="404 Not Found" header=" \"Content-Type\": \"application/json\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"405\"" body=" No response body to print"
level=DEBUG msg=Request id=4 url=http://127.0.0.1:15000/v2/aaaa/busybox/blobs/uploads/ method=POST header=" \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=3 status="202 Accepted" header=" \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"0\"\n \"Location\": \"/v2/aaaa/busybox/blobs/uploads/c40f4bc9-e75e-47b3-b69f-0f45ec4ce74a\"\n \"Range\": \"0-0\"" body=" No response body to print"
level=DEBUG msg=Request id=5 url="http://127.0.0.1:15000/v2/aaaa/busybox/blobs/uploads/c40f4bc9-e75e-47b3-b69f-0f45ec4ce74a?digest=sha256%3Ad7d6aa55bf3d53af143d0a454758d8a7fa4ce4f5e5c4a49c7140e88529e5b6a2" method=PUT header=" \"Content-Type\": \"application/octet-stream\"\n \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=4 status="202 Accepted" header=" \"Location\": \"/v2/aaaa/busybox/blobs/uploads/bed16a67-33d9-4404-ba14-b3c12f6ed803\"\n \"Range\": \"0-0\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"0\"" body=" No response body to print"
level=DEBUG msg=Request id=6 url="http://127.0.0.1:15000/v2/aaaa/busybox/blobs/uploads/bed16a67-33d9-4404-ba14-b3c12f6ed803?digest=sha256%3Af8c71ec7aa5e0cca02e33bbd7678b947bac803b893cb0015cb4707f01fa3721e" method=PUT header=" \"Content-Type\": \"application/octet-stream\"\n \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=6 status="201 Created" header=" \"Content-Length\": \"0\"\n \"Docker-Content-Digest\": \"sha256:f8c71ec7aa5e0cca02e33bbd7678b947bac803b893cb0015cb4707f01fa3721e\"\n \"Location\": \"/v2/aaaa/busybox/blobs/sha256:f8c71ec7aa5e0cca02e33bbd7678b947bac803b893cb0015cb4707f01fa3721e\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"" body=" No response body to print"
level=DEBUG msg=Response id=5 status="201 Created" header=" \"Docker-Content-Digest\": \"sha256:d7d6aa55bf3d53af143d0a454758d8a7fa4ce4f5e5c4a49c7140e88529e5b6a2\"\n \"Location\": \"/v2/aaaa/busybox/blobs/sha256:d7d6aa55bf3d53af143d0a454758d8a7fa4ce4f5e5c4a49c7140e88529e5b6a2\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"0\"" body=" No response body to print"
level=DEBUG msg=Request id=7 url=http://127.0.0.1:15000/v2/aaaa/busybox/manifests/sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d method=PUT header=" \"Content-Type\": \"application/vnd.oci.image.manifest.v1+json\"\n \"User-Agent\": \"Helm/4.0+unreleased\""
level=DEBUG msg=Response id=7 status="201 Created" header=" \"Docker-Content-Digest\": \"sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d\"\n \"Location\": \"/v2/aaaa/busybox/manifests/sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"0\"" body=" No response body to print"
level=DEBUG msg=Request id=8 url=http://127.0.0.1:15000/v2/aaaa/busybox/manifests/sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d method=GET header=" \"User-Agent\": \"Helm/4.0+unreleased\"\n \"Accept\": \"application/vnd.oci.image.manifest.v1+json\""
level=DEBUG msg=Response id=8 status="200 OK" header=" \"Access-Control-Allow-Origin\": \"*\"\n \"Content-Length\": \"593\"\n \"Content-Type\": \"application/vnd.oci.image.manifest.v1+json\"\n \"Docker-Content-Digest\": \"sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"" body="{\"schemaVersion\":2,\"config\":{\"mediaType\":\"application/vnd.cncf.helm.config.v1+json\",\"digest\":\"sha256:f8c71ec7aa5e0cca02e33bbd7678b947bac803b893cb0015cb4707f01fa3721e\",\"size\":147},\"layers\":[{\"mediaType\":\"application/vnd.cncf.helm.chart.content.v1.tar+gzip\",\"digest\":\"sha256:d7d6aa55bf3d53af143d0a454758d8a7fa4ce4f5e5c4a49c7140e88529e5b6a2\",\"size\":4917}],\"annotations\":{\"org.opencontainers.image.created\":\"2025-05-26T12:17:51-06:00\",\"org.opencontainers.image.description\":\"A Helm chart for Kubernetes\",\"org.opencontainers.image.title\":\"busybox\",\"org.opencontainers.image.version\":\"100.100.101\"}}"
level=DEBUG msg=Request id=9 url=http://127.0.0.1:15000/v2/aaaa/busybox/manifests/100.100.101 method=PUT header=" \"User-Agent\": \"Helm/4.0+unreleased\"\n \"Content-Type\": \"application/vnd.oci.image.manifest.v1+json\""
level=DEBUG msg=Response id=9 status="201 Created" header=" \"Docker-Content-Digest\": \"sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d\"\n \"Location\": \"/v2/aaaa/busybox/manifests/sha256:e0a6bd0b52bbdad52bd8952fcc625584261957b23d2faaceaaf94f0f10c5203d\"\n \"Date\": \"Mon, 26 May 2025 18:18:11 GMT\"\n \"Content-Length\": \"0\"" body=" No response body to print"
There may be room for optimization, although this is typical for how the ORAS CLI works with ORAS v2. Push the manifest and then tag it. I'd want to understand why the ORAS CLI works that way although it may be just to support multiple tags.
I agree it should be looked at, but the operation of pushing the manifest twice is not a huge performance hit.
Hi @scottrigby @TerryHowe ,
Thanks for reviewing the issue. could you please help to determine the right approach here:
- Should the goal be to restore a single manifest push as default, or is the current two-push (digest then tag) the intended standard?
- If the two-push is standard, would a flag for an optional single push be a suitable optimization?
- Should we open RFC (Request for Comments) for it before implementation?
thanks!
I think this issue is sufficient to track this. I've been looking at problems where users are broken rather than performance issues. If anyone else has time to evaluate this, that would be appreciated.
Hi @TerryHowe , @scottrigby ,
Just checking in to see if there are any updates on this issue. It’s worth reiterating that this change in behavior, specifically the duplication of uploads when using the Helm CLI to upload an image was not present in versions prior to 3.17.3. Understanding if the two-push behavior is intended or if we can revert to a single manifest push would be greatly appreciated.
Thank you!
Well, I did some investigation, but no coding. I noticed that oras-go has AutoSaveIndex and a SaveIndex() method. Presumably, those two things would do what you want. Just need to set AutoSaveIndex=false and call the SaveIndex() method at the end manually (in theory).
This issue has been marked as stale because it has been open for 90 days with no activity. This thread will be automatically closed in 30 days if no further activity occurs.
![github-actions[bot] avatar](https://avatars.githubusercontent.com/in/15368?v=4 "Published by a pub.dev verified publisher"){kind=small}