[from qiskit] SPSA: occasionally evaluate at unperturbed parameters

[ElePT]

What should we add?

SPSA is an efficient optimizer that approximates the gradient each iteration using only two evaluations of the cost-function f(theta). These evaluations occur at f(theta+delta) and f(theta-delta), where delta is a perturbation along a chosen axis in parameter space.

Notably, the cost function at the unperturbed parameters, f(theta), is not evaluated until the very end of the optimization process. This makes sense, as evaluating it each iteration would make the optimization take roughly 50% longer. However, it blinds the user to how the actual value of the cost function is evolving over time.

I think it would be helpful to have an option to additionally evaluate f(theta) periodically throughout the optimization. E.g. the user could specify a period of 10 iterations, which would only increase the runtime ~5%. I think this could provide useful diagnostic information, both for post-mortem analyses (e.g. tuning hyperparameters) or for real-time decision making (e.g. deciding whether to terminate an unpromising optimization run early to save time). It should also help produce plots that are easier to interpret.

Also, this could in some cases allow users to manually terminate (interrupt) a lengthy optimization run and still get a usable result, e.g. if the user notices that the optimizer has already converged but still has many iterations left to run, they could interrupt the execution and still salvage a good estimate of f(theta_optimal) from the saved data.