Automated Benchmarking for Provider Creation Time

[drew-distefano]

What is the expected feature or enhancement?

Write an automated benchmarking test to measure provider creation time. Creation time is currently at 7-8 seconds (for qiskit-ibm-provider) which is below the optimal 2-3 second creation time defined by @blakejohnson in Qiskit/qiskit-ibm-provider/issues/555, so we want to introduce automation to ensure we notice regressions.

Acceptance criteria

1. Tool accurately measures provider creation time
2. Record the outcome of the benchmark so we can track this metric over time. Consider using ASV to produce a dashboard like what we have for Qiskit.
3. Integrate with current cloud staging tests that are run prior to release

[drew-distefano]

@blakejohnson per your suggestion. Please let me know if you have any edits.

[blakejohnson]

Fail test if creation time falls below a certain threshold

Thanks for opening this. If you are defining pass/fail criteria, then you would fail if you exceed a threshold.

However, what I really want is something that records the outcome of the benchmark so we can track this metric over time. You might consider using ASV to produce a dashboard like what we have for Qiskit. Then it isn't so much about rejecting changes the fail the test once, but to observe that we are actively driving this time lower.

[drew-distefano]

@blakejohnson Thanks for the feedback. I updated the second acceptance criteria to reflect the goal of tracking over time.