Add E2E test for otel collector and target allocator observability

[swiatekm]

We currently don't have an E2E test for checking if spec.observability.metrics.enableMetrics creates ServiceMonitor and PodMonitor definitions which actually allow the metrics to be scraped correctly. Add a test that verifies this. This will require enabling the target allocator so the otel collector can scrape its own metrics, so it may as well check the same setting for target allocator itself.

See https://github.com/open-telemetry/opentelemetry-operator/issues/3990 for how this can be done using chainsaw.

[carsontham]

i will work on this :)

[swiatekm]

@carsontham thank you for taking this up. To give a bit more detail, for collector metrics we just want to check if we can get any metrics at all - it's not a project we own, so we don't have to verify what those are exactly. For the target allocator, we want to check exhaustively, same as we did for the operator. It's fine to do these in separate PRs, but I'll leave that decision up to you. Does that make sense?

[carsontham]

hey @swiatekm ! thanks for the extra details and yes it makes sense. In that case, I will separate the e2e tests into two different PRs :)

[carsontham]

hey @swiatekm ! i have created a PR here for asserting target allocator metrics. Will follow up with another one to test the collector after this PR is approved :)

[carsontham]

hey @swiatekm ! i have added an additional check for collector metrics in the same e2e test with the target allocator here, to check if we get any metrics at all from the collector. thanks !