[Spike] Determine the future of the agent's performance testing

[kaylareopelle]

Performance is very important to the agent. Our code runs within an application's environment. We want to have as light of a footprint as possible, improve our existing performance where we can, and ensure changes don't make the agent run more slowly.


We're uncertain about the future of the Ruby agent's current performance testing suite. 
The suite is prone to confusion, as running the tests multiple times on the same machine produces different results. The percentage difference may be significant for nanoseconds of change. We may gain performance during one run and lose it in the next. This makes the existing solution unreliable and difficult to interpret.

During this spike, the engineer should investigate modern ways of performance testing in Ruby and compare them with our existing suite.


• Understand how our current performance suite works
• Understand modern Ruby performance testing and pre-built options that already exist
• Determine whether we should improve our existing performance suite or adopt a new process
• If we choose to improve our existing suite, decide how we need to improve it and break that into stories
• If we choose to adopt a new process, propose the new process and a migration approach


Any performance suite used by the Ruby agent should have the following qualities:

• 

consistency - same results using the same environment over and over again 

• 

coverage reporting - what's our current performance test coverage? where do we need to add tests?
• 


easy to read results - which results deserve closer attention?
• 



runs on the CI - automatically runs the performance tests weekly and on release







 



Closed, incomplete issues to consider during the investigation: 


• 



https://github.com/newrelic/newrelic-ruby-agent/issues/1395
• https://github.com/newrelic/newrelic-ruby-agent/issues/1860
• https://github.com/newrelic/newrelic-ruby-agent/issues/1861


We've discussed running performance tests within a typical customer environment, such as a Rails application. That type of performance testing could be evaluated during this spike. It may also be seen as out of scope. Choose your own adventure.

[workato-integration[bot]]

https://issues.newrelic.com/browse/NR-150287