Allow for benchmarking with MPI

[abussy]

I've been trying to enable MPI benchmarks on a package I help develop. As things stand, it is seemingly impossible. In this PR, I propose two minor changes that would allow this feature. These amendments are general and do not explicitly depend on MPI.jl. They do not change the behavior of PkgBenchmark in any other case.

1. To run a MPI benchmark, I'd like to pass the following juliacmd to BenchmarkConfig():

juliacmd = `$(mpiexec()) -n $n_ranks $(joinpath(Sys.BINDIR, Base.julia_exename()))`

Because of the way the --color flag is dealt with, the -C, -J and -g1 flags are passed to mpiexec causing an error. Since the orders of the flags does not matter, I propose to simply append --color at the end of juliacmd rather than after the first word.

2. Benchmark results are written to a temporary file, before it is parsed to a JSON dictionary. With MPI, each rank will write to the same file, causing a race condition. To solve this issue, I propose to use atomic writes as inspired by this PR: JuliaLang/Pkg.jl/pull/3235. Each rank will overwrite the same information in turn (assuming a barrier near the end of the application).

Note that each rank logs the benchmark progress, and the terminal can get a little crowded. This is not much of an issue, as the benchmark report is unique and correct.

Thanks for considering this contribution.

[codecov[bot]]

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 91.32%. Comparing base (6aab495) to head (3226e70). Report is 8 commits behind head on master.

Additional details and impacted files

@@            Coverage Diff             @@
##           master     #160      +/-   ##
==========================================
+ Coverage   91.22%   91.32%   +0.10%     
==========================================
  Files           7        6       -1     
  Lines         342      346       +4     
==========================================
+ Hits          312      316       +4     
  Misses         30       30              

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

:rocket: New features to boost your workflow:

• :snowflake: Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[KristofferC]

If this works well for you then it's fine by me.

[abussy]

Hello, would it be possible to merge this PR to make it widely available?