Regularization/generic loss function

[lindonroberts]

trafficstars

Modify DFO-LS to allow different loss functions (not just sum-of-squares), when the analytic form is known, so full model can be built using first derivatives (e.g. currently, have y -> y^2, with first/second derivatives y -> 2y and y -> 2). That is, have generic support for composite functions. Link is "loss" input to scipy.optimize.least_squares.

Also, add ability to add a regularization term (with known structure), e.g. Tikhonov/ridge regression.

[lindonroberts]

Scipy function: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.least_squares.html

[lindonroberts]

A simpler option (also separately useful), would be to have weighted least-squares: either

• f(x) = sum_{i=1}^{m} w_i * r_i(x)^2 or
• f(x) = ||r_i(x)||{A}^2, where ||v||{A} = v' * A * v for A symmetric positive definite