Cannot update multiple images when using local container registry such as localhost:5000

[tkat0]

Hi team. I'd like to report that someone else may have the same problem. I have a local (on-premise) Kubernetes environment on microk8s for testing, where I use argocd-image-updater. If there is anything I can do to help, please let me know.

Describe the bug

I found that argocd-image-updater could not update multiple images under the following conditions. argocd-image-updater updates only one image.

• Using Kustomize
• Update targets are multiple local docker registry images
  • Note that image includes :port such as localhost:5000 or 192.168.0.100:32000
  • I'm using microk8s local registry
  • (Same for local cluster such as minikube)

This is an example Application to cause the problem. Note the argocd-image-updater.argoproj.io/image-list.

apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: myapp
  namespace: argocd
  annotations:
    argocd-image-updater.argoproj.io/image-list: 192.168.0.100:32000/api-server,192.168.0.100:32000/app
    argocd-image-updater.argoproj.io/force-update: 'true'
    argocd-image-updater.argoproj.io/update-strategy: latest
    argocd-image-updater.argoproj.io/write-back-method: git
    argocd-image-updater.argoproj.io/git-branch: main
    argocd-image-updater.argoproj.io/write-back-target: kustomization
spec:
  project: myapp
  source:
    repoURL: [email protected]:tkat0/xxx.git
    targetRevision: HEAD
    path: kubernetes/myapp/development
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
  destination:
    server: https://kubernetes.default.svc
    namespace: default

To Reproduce

It can be reproduced by unit tests. I added the test to pkg/argocd/update_test.go to confirm.

func Test_UpdateApplication(t *testing.T) {
	// Based on Test_UpdateApplication/Test_kustomize_w/_multiple_images_w/_different_registry_w/_different_tags
	t.Run("Test kustomize w/ multiple images w/ local registry w/ different tags", func(t *testing.T) {
		mockClientFn := func(endpoint *registry.RegistryEndpoint, username, password string) (registry.RegistryClient, error) {
			regMock := regmock.RegistryClient{}
			regMock.On("NewRepository", mock.Anything).Return(nil)
			regMock.On("Tags", mock.Anything).Return([]string{"1.0.2", "1.0.3"}, nil)
			return &regMock, nil
		}

		argoClient := argomock.ArgoCD{}
		argoClient.On("UpdateSpec", mock.Anything, mock.Anything).Return(nil, nil)

		kubeClient := kube.KubernetesClient{
			Clientset: fake.NewFakeKubeClient(),
		}
		annotations := map[string]string{
			// Cannot set `:>=1.0.1` due to similar another problem: msg="Using version constraint '5000/foobar:>=1.0.1' when looking for a new tag"
			common.ImageUpdaterAnnotation:                     "foobar=localhost:5000/foobar:>=1.0.1,foobar=localhost:5000/barbar",
			common.ApplicationWideForceUpdateOptionAnnotation: "true",
		}
		appImages := &ApplicationImages{
			Application: v1alpha1.Application{
				ObjectMeta: v1.ObjectMeta{
					Name:        "guestbook",
					Namespace:   "guestbook",
					Annotations: annotations,
				},
				Spec: v1alpha1.ApplicationSpec{
					Source: v1alpha1.ApplicationSource{
						Kustomize: &v1alpha1.ApplicationSourceKustomize{
							// (A) Commenting this out does the same thing.
							Images: v1alpha1.KustomizeImages{
								"localhost:5000/foobar:1.0.1",
								"localhost:5000/barbar:1.0.1",
							},
						},
					},
				},
				Status: v1alpha1.ApplicationStatus{
					SourceType: v1alpha1.ApplicationSourceTypeKustomize,
					Summary: v1alpha1.ApplicationSummary{
						// (B) Commenting this out does the same thing.
						Images: []string{
							"localhost:5000/foobar:1.0.1",
							"localhost:5000/barbar:1.0.1",
						},
					},
				},
			},
			Images: *parseImageList(annotations),
		}
		res := UpdateApplication(&UpdateConfiguration{
			NewRegFN:   mockClientFn,
			ArgoClient: &argoClient,
			KubeClient: &kubeClient,
			UpdateApp:  appImages,
			DryRun:     false,
		}, NewSyncIterationState())
		assert.Equal(t, 0, res.NumErrors)
		assert.Equal(t, 0, res.NumSkipped)
		assert.Equal(t, 1, res.NumApplicationsProcessed)
		assert.Equal(t, 2, res.NumImagesConsidered)
		assert.Equal(t, 2, res.NumImagesUpdated)
		assert.Equal(t, v1alpha1.KustomizeImages(v1alpha1.KustomizeImages{"localhost:5000/foobar:1.0.3", "localhost:5000/barbar:1.0.3"}), appImages.Application.Spec.Source.Kustomize.Images)
	})
}

You can see that the log is fine. However, the updated image is only "localhost:5000/barbar:1.0.3".

$ go test ./... -v -run Test_UpdateApplication/Test_kustomize_w/_multiple_images_w/_local_registry_w/_different_tags

=== RUN   Test_UpdateApplication/Test_kustomize_w/_multiple_images_w/_local_registry_w/_different_tags
time="2023-01-07T07:21:13+09:00" level=debug msg="Considering this image for update" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="setting rate limit to 20 requests per second" prefix="localhost:5000" registry="https://localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Inferred registry from prefix localhost:5000 to use API https://localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Using no version constraint when looking for a new tag" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="found 2 from 2 tags eligible for consideration" image="localhost:5000/foobar:1.0.1"
time="2023-01-07T07:21:13+09:00" level=info msg="Setting new image to localhost:5000/foobar:1.0.3" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=info msg="Successfully updated image 'localhost:5000/foobar:1.0.1' to 'localhost:5000/foobar:1.0.3', but pending spec update (dry run=false)" alias=foobar application=guestbook image_name=foobar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Considering this image for update" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Using no version constraint when looking for a new tag" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="found 2 from 2 tags eligible for consideration" image="localhost:5000/barbar:1.0.1"
time="2023-01-07T07:21:13+09:00" level=info msg="Setting new image to localhost:5000/barbar:1.0.3" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=info msg="Successfully updated image 'localhost:5000/barbar:1.0.1' to 'localhost:5000/barbar:1.0.3', but pending spec update (dry run=false)" alias=foobar application=guestbook image_name=barbar image_tag=1.0.1 registry="localhost:5000"
time="2023-01-07T07:21:13+09:00" level=debug msg="Using commit message: "
time="2023-01-07T07:21:13+09:00" level=info msg="Committing 2 parameter update(s) for application guestbook" application=guestbook
time="2023-01-07T07:21:13+09:00" level=info msg="Successfully updated the live application spec" application=guestbook
    update_test.go:95: 
                Error Trace:    update_test.go:95
                Error:          Not equal: 
                                expected: v1alpha1.KustomizeImages{"localhost:5000/foobar:1.0.3", "localhost:5000/barbar:1.0.3"}
                                actual  : v1alpha1.KustomizeImages{"localhost:5000/barbar:1.0.3", "localhost:5000/barbar:1.0.3"}

Expected behavior

argocd-image-updater updates all images

Additional context

The cause is due to the implementation of argocd.

When comparing Kustomize image names, argocd checks the image name up to first : as an identifier, so it incorrectly recognizes 192.168.0.100:32000/api-server and 192.168.0.100:32000/app as the same image (192.168.0.100).

• func SetKustomizeImage is called for each image
• func (*ApplicationSourceKustomize) MergeImage and func (KustomizeImage) Match overwrote the same image rather than added.

Therefore, you can't use registry names containing colon :. You can use these workarounds.

• Option 1: Publish Local Registry with a URI that doesn't include a port, such as 192.168.0.101.
  • Use LoadBalancer of service type or Ingress
• Option 2: Use GCR, ECR, or other registries.
  • https://argocd-image-updater.readthedocs.io/en/stable/configuration/registries/

Version

• argocd: v2.5.5+fc3eaec
• argocd-image-updater: v0.12.1

Logs

See To Reproduce" section.