test: BatchProcessor stress testing

[cijothomas]

Trying out one way to test overall throughput in batching scenarios. This could be another unit test?

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 0% with 5 lines in your changes missing coverage. Please review.

Project coverage is 79.7%. Comparing base (52cd0e9) to head (56c62fb).

Files with missing lines	Patch %	Lines
opentelemetry-sdk/src/logs/export.rs	0.0%	5 Missing :warning:

Additional details and impacted files

@@           Coverage Diff           @@
##            main   #2788     +/-   ##
=======================================
- Coverage   79.7%   79.7%   -0.1%     
=======================================
  Files        123     123             
  Lines      23136   23141      +5     
=======================================
- Hits       18448   18447      -1     
- Misses      4688    4694      +6     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[scottgerring]

This is a cool idea!

I wonder if ....

• there are similar examples of this sort of thing in other opentelem projects we can be inspired by (probably you already know this! )
• there's an idiomatic way ™️ of fitting this into rust's testing infrastructure. E.g. could we structure it to be more like a criterion test ("it took X ms to push through Y logs"), or is that total madness because of the granularity?

[cijothomas]

This is a cool idea!

I wonder if ....

• there are similar examples of this sort of thing in other opentelem projects we can be inspired by (probably you already know this! )
• there's an idiomatic way ™️ of fitting this into rust's testing infrastructure. E.g. could we structure it to be more like a criterion test ("it took X ms to push through Y logs"), or is that total madness because of the granularity?

Inspired from .NET's https://github.com/open-telemetry/opentelemetry-dotnet/pull/1086 4 years ago!

We do have some tests similar to this in this repo already (metrics focused), and I expect to make a normal-test one for this.

The stress test is to test various additional scenarios, like what happens in network cable is pulled out, what if only single core is available.. what if we run this for a long time etc.

("it took X ms to push through Y logs")

I'll be working a lot on these kind of ideas for next few weeks/months. (A specific example is: Today, BatchProcessor sequentially calls Exporter.export(). To achieve higher throughput, this may not be sufficient. We have spec discussion about improving this, so the tests here will prove/disprove the needs)

[scottgerring]

The stress test is to test various additional scenarios, like what happens in network cable is pulled out, what if only single core is available.. what if we run this for a long time etc.

Sounds useful!

Thinking out loud, it'd be amazing if we could track this over time somehow. I don't know if we could use e.g. Sonarqube or bencher, or some other SaasY code quality platform to do this automatically. Maybe that ties into a better way to track the micro-benchmarks too. We could be clever and run this once a week, say, and then bisect manually if there are big regressions.

[cijothomas]

The stress test is to test various additional scenarios, like what happens in network cable is pulled out, what if only single core is available.. what if we run this for a long time etc.

Sounds useful!

Thinking out loud, it'd be amazing if we could track this over time somehow. I don't know if we could use e.g. Sonarqube or bencher, or some other SaasY code quality platform to do this automatically. Maybe that ties into a better way to track the micro-benchmarks too. We could be clever and run this once a week, say, and then bisect manually if there are big regressions.

I'll steal from what Collector is doing: https://opentelemetry.io/docs/collector/benchmarks/