(stess)test derivatives

[tpapp]

DynamicHMC does not deal well with incorrect deriatives. These can result from AD errors (conceptual errors in mature AD libraries are rare, but numerical corner cases can still happen), or bugs in manually coded derivatives.

There should be functionality to test for these things. Ideally independently of AD, eg with finite differences, possibly using Richardson extrapolation (is there a package for that?).

Decisions to make: should this be part of stresstest, or something else?

[chriselrod]

I do think this would be useful.

The problem I had in the other issue was resolved, and the result of misspecified analytical gradients/Jacobian which was used by AD. The bias was subtle, because it was in the log determinant contribution of a variance parameter in a matrix normal distribution. This can be swamped by the quadratic form when far from the posterior mode (eg, as when just evaluating it at randn(dimension(ℓ))).

Calculus.jl already has a check_gradient function. While check_gradient doesn't use it, they also have taylor finite difference, which looks worth considering.

[tpapp]

I find https://github.com/JuliaDiff/FiniteDifferences.jl excellent for calculating derivatives (explicitly avoiding AD, because very often that's what I want to test), so we could support that.