dregsy
dregsy copied to clipboard
Published
20 hours ago •
xelalexv
xelalexv
ecr: implement multiple credential profile support
This makes cross-account syncing much easier, by allowing configuration of two different credential profiles from your AWS SDK configuration.
I updated the README and implemented an E2E test, which generates a credentials file with two profiles, but using the same access keys.
I also ran a real cross-account copy successfully using my branch.
Result of E2E test:
$ make ISOLATED=y TEST_OPTS+="-v -run=MultiProfile" tests Deleted Images:
untagged: golang@sha256:f6cefbdd25f9a66ec7dcef1ee5deb417882b9db9629a724af8a332fe54e3f7b3
deleted: sha256:272e3f68338f33e2a0bd794e416cb7929de154fcef5d5189566c1c499607bb37
deleted: sha256:eb4d2639c1dcc0d3f9a8575156b59e85a5ab3b9956d5f879b1f4a15e48c7a03a
deleted: sha256:07b990a8e964d8f91336471e0464d5270e6c4b5dc66da7250a5aa013c8e97206
deleted: sha256:2bcd1edc793ec112ae8a710f8c33573e410f4c7f2e0d4440afd0b256cf692226
deleted: sha256:e0746526f645b805c9bb2c895779a9ecef99f425b921e6c0fd6a3a84f8dfa32a
deleted: sha256:c1d02b68d22a9d429b1e5933d12680b2c8be3a513e1476bba6be54b577b419c7
deleted: sha256:883122fdbc326d2ea8477b44b7e82c9d774e0e579e7b6872cbdd067391789006
deleted: sha256:dd5242c2dc8ae9782b73f83281625f45bd6217dc79540e1019d5da0913b491b0
Total reclaimed space: 802.8MB
dregsy-test-registry
74372c9bce97f997a0bb95ea36a51b10efabe0f98c37f6ca3e0ea1427c77dea1
testing using alpine-based image:
=== RUN TestE2ESkopeoECRMultiProfile
INFO[0000] dregsy
INFO[0001] skopeo version 1.3.1 commit: 1bb50ac33996b2fbf512d82f48d7ca2931bd0eb4
INFO[0001] relay ready relay=skopeo
INFO[0001] syncing task source=registry.hub.docker.com target=258273616434.dkr.ecr.eu-central-1.amazonaws.com task=test-ecr-multi-profile-pre
INFO[0001] mapping from=/library/busybox to=/dregsy/test
INFO[0001] refreshing credentials registry=registry.hub.docker.com
INFO[0001] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0001] target already exists ref=258273616434.dkr.ecr.eu-central-1.amazonaws.com/dregsy/test
DEBU[0001] verbatim tags: [1.29.2 1.29.3 latest]
INFO[0001] syncing tag tag=1.29.2
DEBU[0003] Getting image source signatures
DEBU[0003] Copying blob sha256:8c5a7da1afbc602695fcb2cd6445743cec5ff32053ea589ea9bd8773b7068185
DEBU[0004] Copying config sha256:e1ddd7948a1c31709a23cc5b7dfe96e55fc364f90e1cebcde0773a1b5a30dcda
DEBU[0004] Writing manifest to image destination
DEBU[0004] Storing signatures
INFO[0004] syncing tag tag=1.29.3
DEBU[0006] Getting image source signatures
DEBU[0007] Copying blob sha256:b4a6e23922ddc3d105fee9afff80151a13fe058143351a8e9294286575f2f37e
DEBU[0007] Copying config sha256:758ec7f3a1ee85f8f08399b55641bfb13e8c1109287ddc5e22b68c3d653152ee
DEBU[0007] Writing manifest to image destination
DEBU[0007] Storing signatures
INFO[0007] syncing tag tag=latest
DEBU[0009] Getting image source signatures
DEBU[0009] Copying blob sha256:8ec32b265e94aafb0d43ab71f1d8f786122c19afb37d25532aea169f414f8881
DEBU[0010] Copying config sha256:42b97d3c2ae95232263a04324aaf656dc80e7792dee6629a9eff276cdfb806c0
DEBU[0010] Writing manifest to image destination
DEBU[0010] Storing signatures
DEBU[0010] task starts ticking task=test-ecr-multi-profile
DEBU[0010] sending initial fire task=test-ecr-multi-profile
INFO[0010] waiting for next sync task...
INFO[0010] syncing task source=258273616434.dkr.ecr.eu-central-1.amazonaws.com target=258273616434.dkr.ecr.eu-central-1.amazonaws.com task=test-ecr-multi-profile
INFO[0010] mapping from=/dregsy/test to=/dregsy/test
INFO[0010] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0010] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0010] target already exists ref=258273616434.dkr.ecr.eu-central-1.amazonaws.com/dregsy/test
DEBU[0010] verbatim tags: [1.29.2 1.29.3 latest]
INFO[0010] syncing tag tag=1.29.2
DEBU[0011] Getting image source signatures
DEBU[0011] Copying blob sha256:8c5a7da1afbc602695fcb2cd6445743cec5ff32053ea589ea9bd8773b7068185
DEBU[0011] Copying config sha256:e1ddd7948a1c31709a23cc5b7dfe96e55fc364f90e1cebcde0773a1b5a30dcda
DEBU[0012] Writing manifest to image destination
DEBU[0012] Storing signatures
INFO[0012] syncing tag tag=1.29.3
DEBU[0012] Getting image source signatures
DEBU[0013] Copying blob sha256:b4a6e23922ddc3d105fee9afff80151a13fe058143351a8e9294286575f2f37e
DEBU[0013] Copying config sha256:758ec7f3a1ee85f8f08399b55641bfb13e8c1109287ddc5e22b68c3d653152ee
DEBU[0013] Writing manifest to image destination
DEBU[0013] Storing signatures
INFO[0013] syncing tag tag=latest
DEBU[0014] Getting image source signatures
DEBU[0014] Copying blob sha256:8ec32b265e94aafb0d43ab71f1d8f786122c19afb37d25532aea169f414f8881
DEBU[0014] Copying config sha256:42b97d3c2ae95232263a04324aaf656dc80e7792dee6629a9eff276cdfb806c0
DEBU[0014] Writing manifest to image destination
DEBU[0015] Storing signatures
INFO[0015] waiting for next sync task...
INFO[0015] TEST - shutting down dregsy
INFO[0015] shutdown flagged, stopping ...
DEBU[0015] stopping tasks
DEBU[0015] task exited task=test-ecr-multi-profile-pre
DEBU[0015] task exiting task=test-ecr-multi-profile
DEBU[0015] task exited task=test-ecr-multi-profile
INFO[0015] all done
DEBU[0015] exit main
INFO[0015] TEST - dregsy stopped
INFO[0015] TEST - validating result
INFO[0015] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0015] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
--- PASS: TestE2ESkopeoECRMultiProfile (15.89s)
PASS
coverage: 33.6% of statements in ./...
ok github.com/xelalexv/dregsy/cmd/dregsy 15.894s coverage: 33.6% of statements in ./...
? github.com/xelalexv/dregsy/internal/pkg/auth [no test files]
? github.com/xelalexv/dregsy/internal/pkg/registry [no test files]
? github.com/xelalexv/dregsy/internal/pkg/relays/docker [no test files]
? github.com/xelalexv/dregsy/internal/pkg/relays/skopeo [no test files]
testing: warning: no tests to run
PASS
coverage: 1.0% of statements in ./...
ok github.com/xelalexv/dregsy/internal/pkg/sync 0.008s coverage: 1.0% of statements in ./... [no tests to run]
? github.com/xelalexv/dregsy/internal/pkg/tags [no test files]
? github.com/xelalexv/dregsy/internal/pkg/test [no test files]
? github.com/xelalexv/dregsy/internal/pkg/test/registries [no test files]
? github.com/xelalexv/dregsy/internal/pkg/util [no test files]
coverage report is in _build/coverage-alpine.html
Total reclaimed space: 0B
dregsy-test-registry
1eb20fe21825252e7118198f8d8445d58a2325ed37292624eca784c7af5d0b2d
testing using ubuntu-based image:
=== RUN TestE2ESkopeoECRMultiProfile
INFO[0000] dregsy
INFO[0001] skopeo version 1.3.0
INFO[0001] relay ready relay=skopeo
INFO[0001] syncing task source=registry.hub.docker.com target=258273616434.dkr.ecr.eu-central-1.amazonaws.com task=test-ecr-multi-profile-pre
INFO[0001] mapping from=/library/busybox to=/dregsy/test
INFO[0001] refreshing credentials registry=registry.hub.docker.com
INFO[0001] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0001] target already exists ref=258273616434.dkr.ecr.eu-central-1.amazonaws.com/dregsy/test
DEBU[0001] verbatim tags: [1.29.2 1.29.3 latest]
INFO[0001] syncing tag tag=1.29.2
DEBU[0002] Getting image source signatures
DEBU[0003] Copying blob sha256:8c5a7da1afbc602695fcb2cd6445743cec5ff32053ea589ea9bd8773b7068185
DEBU[0003] Copying config sha256:e1ddd7948a1c31709a23cc5b7dfe96e55fc364f90e1cebcde0773a1b5a30dcda
DEBU[0004] Writing manifest to image destination
DEBU[0004] Storing signatures
INFO[0004] syncing tag tag=1.29.3
DEBU[0006] Getting image source signatures
DEBU[0006] Copying blob sha256:b4a6e23922ddc3d105fee9afff80151a13fe058143351a8e9294286575f2f37e
DEBU[0006] Copying config sha256:758ec7f3a1ee85f8f08399b55641bfb13e8c1109287ddc5e22b68c3d653152ee
DEBU[0007] Writing manifest to image destination
DEBU[0007] Storing signatures
INFO[0007] syncing tag tag=latest
DEBU[0009] Getting image source signatures
DEBU[0009] Copying blob sha256:8ec32b265e94aafb0d43ab71f1d8f786122c19afb37d25532aea169f414f8881
DEBU[0009] Copying config sha256:42b97d3c2ae95232263a04324aaf656dc80e7792dee6629a9eff276cdfb806c0
DEBU[0010] Writing manifest to image destination
DEBU[0010] Storing signatures
DEBU[0010] task starts ticking task=test-ecr-multi-profile
DEBU[0010] sending initial fire task=test-ecr-multi-profile
INFO[0010] waiting for next sync task...
INFO[0010] syncing task source=258273616434.dkr.ecr.eu-central-1.amazonaws.com target=258273616434.dkr.ecr.eu-central-1.amazonaws.com task=test-ecr-multi-profile
INFO[0010] mapping from=/dregsy/test to=/dregsy/test
INFO[0010] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0010] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0010] target already exists ref=258273616434.dkr.ecr.eu-central-1.amazonaws.com/dregsy/test
DEBU[0010] verbatim tags: [1.29.2 1.29.3 latest]
INFO[0010] syncing tag tag=1.29.2
DEBU[0010] Getting image source signatures
DEBU[0011] Copying blob sha256:8c5a7da1afbc602695fcb2cd6445743cec5ff32053ea589ea9bd8773b7068185
DEBU[0011] Copying config sha256:e1ddd7948a1c31709a23cc5b7dfe96e55fc364f90e1cebcde0773a1b5a30dcda
DEBU[0011] Writing manifest to image destination
DEBU[0011] Storing signatures
INFO[0011] syncing tag tag=1.29.3
DEBU[0012] Getting image source signatures
DEBU[0012] Copying blob sha256:b4a6e23922ddc3d105fee9afff80151a13fe058143351a8e9294286575f2f37e
DEBU[0012] Copying config sha256:758ec7f3a1ee85f8f08399b55641bfb13e8c1109287ddc5e22b68c3d653152ee
DEBU[0012] Writing manifest to image destination
DEBU[0013] Storing signatures
INFO[0013] syncing tag tag=latest
DEBU[0013] Getting image source signatures
DEBU[0013] Copying blob sha256:8ec32b265e94aafb0d43ab71f1d8f786122c19afb37d25532aea169f414f8881
DEBU[0014] Copying config sha256:42b97d3c2ae95232263a04324aaf656dc80e7792dee6629a9eff276cdfb806c0
DEBU[0014] Writing manifest to image destination
DEBU[0014] Storing signatures
INFO[0014] waiting for next sync task...
INFO[0014] TEST - shutting down dregsy
INFO[0014] shutdown flagged, stopping ...
DEBU[0014] stopping tasks
DEBU[0014] task exited task=test-ecr-multi-profile-pre
DEBU[0014] task exiting task=test-ecr-multi-profile
DEBU[0014] task exited task=test-ecr-multi-profile
INFO[0014] all done
DEBU[0014] exit main
INFO[0015] TEST - dregsy stopped
INFO[0015] TEST - validating result
INFO[0015] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
INFO[0015] refreshing credentials registry=258273616434.dkr.ecr.eu-central-1.amazonaws.com
--- PASS: TestE2ESkopeoECRMultiProfile (16.31s)
PASS
coverage: 33.6% of statements in ./...
ok github.com/xelalexv/dregsy/cmd/dregsy 16.319s coverage: 33.6% of statements in ./...
? github.com/xelalexv/dregsy/internal/pkg/auth [no test files]
? github.com/xelalexv/dregsy/internal/pkg/registry [no test files]
? github.com/xelalexv/dregsy/internal/pkg/relays/docker [no test files]
? github.com/xelalexv/dregsy/internal/pkg/relays/skopeo [no test files]
testing: warning: no tests to run
PASS
coverage: 1.0% of statements in ./...
ok github.com/xelalexv/dregsy/internal/pkg/sync 0.009s coverage: 1.0% of statements in ./... [no tests to run]
? github.com/xelalexv/dregsy/internal/pkg/tags [no test files]
? github.com/xelalexv/dregsy/internal/pkg/test [no test files]
? github.com/xelalexv/dregsy/internal/pkg/test/registries [no test files]
? github.com/xelalexv/dregsy/internal/pkg/util [no test files]
coverage report is in _build/coverage-ubuntu.html
@danielkza Thanks for this great PR! And sorry it took me so long to react! Unfortunately, dregsy is just a private side project, and I don't get as much time to work on it as I would like.
First a few top level comments:
general:
- The PR currently addresses two separate concerns - AWS multi-profile & auth auto-refresh based on expiration date supplied in token. I would like to push back the latter. While reviewing, I found that the way how auto refresh is configured needs to be revised. I will possibly deprecate the
auth-refreshinterval setting and handle refreshing in a way that's consistent for ECR and GCR. So if it's not asking to much, could you remove the according changes from this PR?
doc:
- When running dregsy containerized, an AWS credentials file with the profiles needs to be mounted into the container. This should be pointed out in the ECR section of the readme.
tests:
-
e2e/_ecr-credentials.tmplis missing - a test case for the Docker relay is missing
misc:
- Please squash your commits.
- Keep existing white space as is, e.g. don't remove blank lines. Also break lines at approx. 80 characters.
I'll add a few more comments inline in the diff in the next 30 minutes or so.
So if it's not asking to much, could you remove the according changes from this PR?
sure
his should be pointed out in the ECR section of the readme.
sure, but I expected advanced users dealing with multiple profiles to understand this is a prerequisite
e2e/_ecr-credentials.tmpl is missing
my bad, will add it
a test case for the Docker relay is missing
is the choice of relay relevant for this functionality? I guessed it was not, but I can of course be completely mistaken
Keep existing white space as is, e.g. don't remove blank lines.
I removed one blank line that was inconsistent with the rest of the project (after a function signature), and one accidentally as part of editing. Do you really want me to re-add them? seems pedantic, and not in a "keep consistency" sort of way (as I would be re-adding an inconsistency).
Please squash your commits.
You can squash when merging, otherwise I can't keep a history and merge this back from a private fork.
a test case for the Docker relay is missing
is the choice of relay relevant for this functionality? I guessed it was not, but I can of course be completely mistaken
It's probably not, but without testing for both, we cannot say for sure.
Keep existing white space as is, e.g. don't remove blank lines.
I removed one blank line that was inconsistent with the rest of the project (after a function signature), and one accidentally as part of editing. Do you really want me to re-add them? seems pedantic, and not in a "keep consistency" sort of way (as I would be re-adding an inconsistency).
Actually, the convention is no blank line after function signature for fairly short functions, blank line for others. But it's really not a problem, and yes, much to the chagrin of people around me, I am quite pedantic... :smile: