K8SSAND-1628 ⁃ More E2E tests for Prometheus integrations

[Miles-Garnsey]

What is missing?

More E2E tests for Prometheus integration.

Why do we need it?

Issue #597 identified that we were double scraping Prometheus metrics because we were picking up two separate CassDC attached services.

While the immediate bug is resolved under PR #610 it did highlight that we lack E2E tests which fully exercise the Prometheus integration's we've built.

The current tests only confirm that the SMs are deployed. They do not test that the various metrics we expect to see are indeed present, or that they are flowing through into Prometheus in the way we expect.

┆Issue is synchronized with this Jira Story by Unito

[jsanda]

We have the Test_NewCassServiceMonitor_SUCCESS unit test (see here) which only performs this check against the created ServiceMonitor:

assert.Equal(t, "prometheus", actualSM.Spec.Endpoints[0].Port)

Shouldn't the test verify the full state, including both metadata and the spec?

We also have the createSingleDatacenter integration test which checks that the ServiceMonitor is created and deleted (see here). We should probably do additional verification here. We should also have coverage in a multi-DC/cluster test.

[sync-by-unito[bot]]

➤ Miles Garnsey commented:

Probably no longer relevant.