how to run the test suite for DynamicHMC

[tpapp]

I am testing some new heuristics for DynamicHMC. What's the best way to run the test suite on my local machine for a local branch of DynamicHMC? (Sorry if this is obvious in the docs, could not find it).

[itsdfish]

Hi Tamas. No problem. Evidently, I had started to implement a function to run all benchmarks but forgot to finish. I pushed some changes to master with the run_all_benchmarks() function. Currently, the benchmarks will not run the Hierarchical Poisson and LBA for DynamicHMC because I still haven't been able to figure out the numerical issues.

In either case, it will output the results to the Examples/Model/results directory. It should generate a set of plots and put all of the results and metadata into a timestamped folder. If you have any questions or suggestions, let me know.

[itsdfish]

I should also note that Turing 0.7.1 fails because of a change to their internal diagnostics and MCMCChains. It will run from the master branch. Hopefully, they update that soon. If you don't want to do that, you can comment out Turing in each individual example file:

#Linear_Regression_Example.jl

samplers=(
  CmdStanNUTS(CmdStanConfig, ProjDir),
  #AHMCNUTS(AHMCregression, AHMCconfig),
  DHMCNUTS(sampleDHMC)
)

My apologies for these issues. Unfortunately, things can be unstable while the ecosystem adapts to changes.

[tpapp]

Thanks. Is it possible to just run the DynamicHMC benchmarks? (If feasible, I would prefer to do this on my laptop.) Also, I am after correctness, not speed; I am tweaking the initial adaptation heuristics.

[itsdfish]

Yeah. The tests should run locally.

Currently, MCMCBenchmarks is configured to run a set of predefined benchmarks via run_all_benchmarks(). However, you can define your own benchmark suite based on your own needs. I added a folder Examples.zip where I modified the Gaussian_Example.jl file to exclude Turing and CmdStan. I also added Nchains = 2, so that you can get valid r hat estimates (if you want that).

options = (Nsamples=2000, Nadapt=1000, delta=.8, Nd=Nd, Nchains=2)

In that same folder, I modified primary_plots.jl to include some plots that you may be interested in, such as rhat, effective sample size, and parameter recovery. The results based on my machine are in Examples/Gaussian/results.

Once you have configured the Example and primary plots files for each model you want to run, can run something like this:

function run_all_benchmarks(models)
    res = map(models) do m
        @eval module $(Symbol("Test_", m))
            include($m)
        end
    end
 end

where the argument models is a list of paths to your main benchmark files (e.g. Gaussian_Example.jl). Each benchmark will be ran in a separate module to avoid naming conflicts.

[itsdfish]

P.S. I will add the above method of run_all_benchmarks to the package.

[itsdfish]

I'm not sure what happened. The run_all_benchmarks functions stopped working. I think the problem is that the module is not available on all workers. Do you have any suggestions for running the benchmarks in separate modules programmatically?

[goedman]

Hi Tamas,

A stand alone test of LBA is here. LBA.jl is the original version. I have added in that directory a stresstest version and a version that creates an MCMCChains.Chains object.

Thanks for all your help.

Rob

[tpapp]

I could not make it work since it loads a Turing before https://github.com/TuringLang/Turing.jl/pull/960 which is incompatible with current DynamicHMC.

If someone has the time, it would be great if you could run it with DynamicHMC#master.

[itsdfish]

It works with DynamicHMC #master and Turing #master.