Pathfinder.jl Benchmark with posteriordb models

We should benchmark the method similarly to the Pathfinder paper by running Pathfinder on many posteriordb models using code adapted from https://github.com/mlcolab/Pathfinder.jl/issues/94#issuecomment-1270759033. We shouldn't do this in CI though, because the benchmarks would probably take a long time to run, but it would be nice to show the results in the docs and periodically re-run them.

We should look at how other packages like some in the SciML ecosystem do this.

Oct 07 '22 14:10 sethaxen

Now that PosteriorDB.jl has been registered, and BridgeStan is included in StanSample.jl, it should be quite straightforward to set up a Julia script that generates samples from posteriordb models using Pathfinder.

If we want the analysis to be similar to the Pathfinder paper, we should use 1-Wasserstein (aka earth mover) distance to compare the Pathfinder draws to the reference posterior draws in posteriordb, so we should see if this is available in the Julia ecosystem for multidimensional draws.

Oct 28 '22 20:10 sethaxen

A number of benchmarks were performed at https://github.com/mlcolab/PathfinderBenchmarks.jl

Jun 23 '23 10:06 sethaxen