Add usage of benchmark runner

[rashidnhm]

Context: The benchmarks workflow now use the self-hosted benchmark runner

Description of the Change: The GitHub hosted runner is not very consistent with collecting accurate times. A dedicated benchmarking runner has been deployed in AWS and will now be used to run benchmarks.

Benefits: Consistent times.

Possible Drawbacks: Slower overall runtime as it can only process 1 job at a time.

Related GitHub Issues:

[github-actions[bot]]

Hello. You may have forgotten to update the changelog! Please edit doc/releases/changelog-dev.md with:

• A one-to-two sentence description of the change. You may include a small working example for new features.
• A link back to this PR.
• Your name (or GitHub username) in the contributors section.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 99.67%. Comparing base (1bb10be) to head (6541686). Report is 1 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #5103      +/-   ##
==========================================
- Coverage   99.68%   99.67%   -0.01%     
==========================================
  Files         402      402              
  Lines       37534    37246     -288     
==========================================
- Hits        37414    37125     -289     
- Misses        120      121       +1     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[rashidnhm]

Hey @rashidnhm, thank you for that. This will be very important to keep track of PennyLane performance. I can see that we are running the benchmarks suite in a dedicated runner in AWS, but is this always the same machine, or are they equivalent machines? I'm asking that, because analyzing the last three runs, I can see that results are less erratic but still not very consistent. You can see that by comparing the images inside of artifacts with same name in different runs. Also, I can see that we are getting several warnings regarding Node.js 16 actions being deprecated. Can we have actions/checkout@v3, actions/cache@v3, actions/setup-python@v4, actions/upload-artifact@v3, upgraded to actions/checkout@v4, actions/cache@v4, actions/setup-python@v5, actions/upload-artifact@v4?

Hey Amintor, they are running on the exact same instance each time. It is a long running EC2 instance.

As for the deprecation warnings, that is a GitHub raised warning as the actions we use depend on a version of node that's need deprecated. Look at this run of our test suite run on the master branch on github hosted runners. The same warnings are there. These warnings should be addressed but are out of scope for this work.

[AmintorDusko]

Hey @Rashid, just flying by, I didn't check if you did it already, but it would be great if you could clear the reference-benchmarks cache and re-run the CIs to compare results from the same runner. Thanks.

[rashidnhm]

Sure

[trbromley]

Closing as this is no longer required.