Test reporting of data in e2e Instrumentation tests

[pavolloffay]

The e2e instrumentation tests verify the structure of the workload deployment (if all env vars are set etc.). However the test should execute an endpoint on the test app that will trigger span generation and then the test should verify if the span arrived to the collector or some mock storage.

@anuraaga the integration tests in the java auto-instrumentation are dosing something similar, is there something we could reuse from that project - e.g. the mock storage?

[anuraaga]

I think this might be a good segway into filing an OTEP or something for a project-wide black-box instrumentation testing suite. I can't think of anything we would implement in an e2e test that wouldn't apply to every language at the same time - sending a request, verifying sent OTLP is a very common concern.

[jpkrohling]

then the test should verify if the span arrived to the collector or some mock storage

At the collector would be easy, as we have metrics already with the number of spans seen by each component, IIRC.

[pavolloffay]

I would prefer to get the reported data. This is what the java auto-instrumentation tests do. The SDK reports to a collector and the collector reports to a service with in-memory storage that can be queried.

[jaronoff97]

@pavolloffay what's the load to make this work? We now do something similar with the target allocator e2e tests