Add automated testing against well known registries on commits and releases

[joaopapereira]

Acceptance Criteria:

As a contributor to imgpkg I want to have confidence that my changes did not introduce inconsistent behavior on different registries.

When a new commit is pushed to imgpkg, I expect to see github actions running integration tests on:

• [x] Harbor
• [x] GCR

When a new release of imgpkg is cut, I expect to see github actions running integration tests on:

• [x] Harbor
• [x] GCR
• [ ] https://cloud.google.com/artifact-registry
• [ ] DockerHub

Notes:

• Since DockerHub has rate limiting, running the integration tests only during a release is sufficient.
• For the integration with harbor, Joao found that we could stand up an instance using https://github.com/bitnami/bitnami-docker-harbor-portal/blob/master/docker-compose.yml, so maybe we could just start a container to do this.
• Feel free to tackle this work all at once, or one registry at a time using 3 different PRs.
  • in fact, if supporting GCR and DockerHub is not "just" an incremental investment, consider whether it makes sense to tackle those registries, later.

👇🏻 Original request👇🏻

---

• Describe the problem/challenge you have As a user, I want to be sure that imgpkg works with different registries

---

Describe the solution you'd like When testing imgpkg using Github action, we should try to test against a local registry and against more widely spread registries, like gcr, harbor, and dockerhub.

Anything else you would like to add: I spun up this issue from #131, thanks for the idea @DennisDenuto For this issue, we should do the following:

• [x] Run integration tests against a gcr instance
• [x] Run integration tests against an Harbor instance
  • Is there a Harbor instance that we can use?
  • Should we instantiate a Harbor instance and use it for testing?
  • Can we run Harbor as a container similar to what we are doing with the registry:2?
• [ ] Run integration tests against a dockerhub, we should be careful about this due to rate limiting. Maybe we could consider running these tests as part of a Release process

---

Vote on this request

This is an invitation to the community to vote on issues, to help us prioritize our backlog. Use the "smiley face" up to the right of this comment to vote.

👍 "I would like to see this addressed as soon as possible" 👎 "There are other more important things to focus on right now"

We are also happy to receive and review Pull Requests if you want to help working on this issue.

[joaopapereira]

For the integration with harbor, I found that we could stand up an instance using https://github.com/bitnami/bitnami-docker-harbor-portal/blob/master/docker-compose.yml so maybe we could just start a container to do this.

I think we can split this issue into 3 if we believe this is too much work or want to tackle this at different times, or even create these in 3 different PR associated with this same issue.

[gcheadle-vmware]

Acceptance Criteria:

As a contributor to imgpkg I want to have confidence that my changes did not introduce inconsistent behavior on different registries.

When a new commit is pushed to imgpkg, I expect to see github actions running integration tests on:

• [ ] Harbor
• [ ] GCR

When a new release of imgpkg is cut, I expect to see github actions running integration tests on:

• [ ] Harbor
• [ ] GCR
• [ ] DockerHub

---

Notes:

• Since DockerHub has rate limiting, running the integration tests only during a release is sufficient.
• For the integration with harbor, Joao found that we could stand up an instance using https://github.com/bitnami/bitnami-docker-harbor-portal/blob/master/docker-compose.yml, so maybe we could just start a container to do this.
• Feel free to tackle this work all at once, or one registry at a time using 3 different PRs.

(edit by @pivotaljohn: moved Harbor to the top of this list; see comment for context.)

[pivotaljohn]

We now have experienced two regressions that were due to Harbor-specific behavior:

• Imgpkg fails to pull a bundle if a registry does not support the HEAD call. #131
• imgpkg fails if location image pull is denied instead of not found #166

Given that this is becoming a more and more common use-case (i.e. Carvel tools against Harbor), the probability of this risk is becoming less theoretical and more probable.

@gcheadle-vmware spec'ed:

Feel free to tackle this work all at once, or one registry at a time using 3 different PRs.

Agreed. In fact, we may find that we'll want to split this work into three separate stories if the incremental effort to support the 2nd and 3rd registries isn't relatively small.

Let's target Harbor first, given our experience.

[pivotaljohn]

Marking for bump in priority given we've had multiple issues and the impact it has on downstream integrators/products ==> "recommended"

[DennisDenuto]

Let's target Harbor first, given our experience

Our past experiences were with harbor, but indirectly accessed via TanzuNet Registry. So I would propose we first test with TanzuNet Registry since there are some 'rules' overlayed on top of Harbor that has been known to be 'differences' that caused issues in the past.

[aaronshurley]

Consider how we want to clean up any testing artifacts.

[DennisDenuto]

Added google's new container registry into the list https://cloud.google.com/artifact-registry