Add Variational inference

[jgrashorn]

Added variational inference in the form of maximum likelihood and maximum a posteriori estimates. Also wrote some tests and added examples in the bayesianupdating demo.

[codecov[bot]]

Codecov Report

Attention: Patch coverage is 84.90566% with 8 lines in your changes missing coverage. Please review.

Project coverage is 94.04%. Comparing base (5c6faaa) to head (f00096f). Report is 18 commits behind head on master.

Files with missing lines	Patch %	Lines
src/modelupdating/bayesianMAP.jl	84.90%	8 Missing :warning:

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #203      +/-   ##
==========================================
- Coverage   94.35%   94.04%   -0.31%     
==========================================
  Files          35       36       +1     
  Lines        1577     1630      +53     
==========================================
+ Hits         1488     1533      +45     
- Misses         89       97       +8     

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

[jgrashorn]

Comments have been implemented. Also changed the approximations to be able to handle multi-modal distributions when the user supplies more than one initial point. Both methods have been added to the existing model updating example.

[jgrashorn]

Added file for point estimates, renamed all types with a -Bayesian suffix for easier identification. Moved the optimization and the setup into a function taking an AbstractBayesianPointEstimate argument. Implemented the existing binomial tests for the point estimation methods. Added (optional) box constraints to optimization since they are sometimes needed for bounded variables (e.g. in Beta distribution). Documentation is still needed.

[jgrashorn]

Wrote some documentation, might still be lacking, but there are now one example each in docs/literate/bayesianupdating and in the manual, with some explanation.

[jgrashorn]

I also find the naming "point estimates" slightly odd. I guess you wanted to generalised "maximum likelihood estimation" and "MAP"?

I am also not happy about the naming, but couldn't come up with a better name to summarize what MLE and MAP are doing.

[AnderGray]

I am also not happy about the naming, but couldn't come up with a better name to summarize what MLE and MAP are doing.

Thanks for the fixes. Maybe we could just pick one, either max likelihood or MAP, and use it for naming the files/titles. I swing towards MAP cause then it's a Bayesian method. It should be quite clear that if we can do one, we can do the other. Maybe better understood than "Bayesian point estimates".

In the docstrings, you can also link stuff, so people can find them:

I.e. at the end of the MAP docstrings add

See also [`MaximumLikelihoodBayesian`](@ref), [`bayesianupdating `](@ref),  [`TransitionalMarkovChainMonteCarlo`](@ref).

And at the end of max likelihood docstrings add:

See also [`MaximumAPosterioriBayesian`](@ref), [`bayesianupdating `](@ref),  [`TransitionalMarkovChainMonteCarlo`](@ref).

[AnderGray]

Thanks for the changes, good from my side 👍

[FriesischScott]

Thanks everyone! We can always revisit the naming later on.