Clarify (and improve) testing reliability

[mkilchhofer]

We should have a look into multiple parts of testing:

• Acceptance tests sometimes fail (relatively often)
• Against what versions of Argo CD should we test (currently we test against EOL versions of Argo CD)
• internal/provider/data_source_application_test.go why do we specify a version constraint here? Can we remove it?
• ..? (list not complete)

[onematchfox]

Against what versions of Argo CD should we test (currently we test against EOL versions of Argo CD)

Per https://github.com/argoproj-labs/terraform-provider-argocd?tab=readme-ov-file#compatibility-promise this should be (yes, the test config is out of date :smile:):

This provider is compatible with at least the last 2 minor releases of ArgoCD (e.g, ranging from 1.(n).m, to 1.(n-1).0, where n is the latest available minor version).

internal/provider/data_source_application_test.go why do we specify a version constraint here? Can we remove it?

I'll need to rack my brain/go through old commit history on this one. There was a reason but I can't for the life of me remember why off the top of my head.

[the-technat]

This provider is compatible with at least the last 2 minor releases of ArgoCD (e.g, ranging from 1.(n).m, to 1.(n-1).0, where n is the latest available minor version).

So for my understanding, this would currently be 2.12 and 2.11?

[onematchfox]

So for my understanding, this would currently be 2.12 and 2.11?

Correct. That's been the "promise" until now. Although, generally we would test the last 3 versions via the config here.

[the-technat]

I think we should also add different Terraform versions to our test matrix, ensuring it works when not using the latest Terraform version.

[github-actions[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

[the-technat]

We already did some work on this:

• #568
• #566
• #565
• #563
• #559

Only open topic for me would be to write down when/how we update the Argo-CD client lib + k8s version for the CI cluster.