Incremental saving of dynesty results

[bd-j]

Like for the emcee fitting, we should have an option for periodically saving the dynesty results to disk during a run (the current setup already uses the dynesty sampling iterator) to guard against cancelled jobs and help in debugging. Also should probably keep the output from flushing stderr, since this overwrites other diagnostic info.

[jrleja]

Also it would be better to create the h5 file only when flushing sampling results to disk, whether that's using the periodic flushing to disk or at the end of the sampling, to avoid creating empty h5 files if a run is aborted.

I'll get on this as soon as dynesty has restart capability.

[themiyan]

when I run the dynasty sampler I end up with multiple h5 files for the same run, e.g.,

#%id_%timestamp%_dynesty_mcmc.h5
26_1575641926_dynesty_mcmc.h5
26_1575641927_dynesty_mcmc.h5
26_1575641928_dynesty_mcmc.h5

I guess this behaviour is due to periodic saving of results? In this case I assume I should only use the latest h5 result to investigate the outputs?

[bd-j]

The sampling with dynesty does not currently implement incremental writes during sampling (which anyway would be to the same file). This looks like somehow multiple fits were run nearly at once, maybe on separate cores?

[themiyan]

aha maybe its to do with the MPI then, would this work?

mpirun -np 4 python params_dynesty_test.py --objid=20 --dynesty

or MPI should be run in the old way? e.g.,

mpirun -np 4 python prospector_dynesty.py –param_file=params_dynesty_test.py --objid=20

[themiyan]

I've opened a new issue #160 for this.

[JohannesBuchner]

You could address this via #201, which would enable incremental saving and efficient MPI parallelisation.