Add tests that check for convergence between NiMARE and other packages

[tyarkoni]

Rather than try to get very clever about checking the values that come out of NiMARE, a simple way to ensure we're in the right ballpark would be to run some of our test datasets through existing implementations of ALE, MKDA, etc., get the output images, and include them in our test data. Then we can have a suite of tests that check that the p, z, etc. maps we produce look sufficiently similar to results from other tools.

I'll feel a lot better once we have something like that, as it will allow us to fiddle around with the algorithms without having to worry about silent errors that change results but still produce images.

[tsalo]

I agree, although what do we do when there are differences?

• I know I fixed some minor bugs in Simon's MATLAB code a few years back, which is what I used to create the ALE Python implementation. Even then, the MATLAB code might differ from the C++ (or C?) code from the GingerALE GUI.
• The MKDADensity implementation is primarily conceptual, since the MKDA toolbox is massive, confusing, and in MATLAB.
• SCALE is also conceptual since there are no implementations out in the world, AFAIK.
• MKDAChi2 is easy enough to compare, since I got it from Neurosynth.
• ALESubtraction will differ from the official version, since we consciously changed the approach.

[tyarkoni]

That's a fair point, but I'm not talking about checking for convergence to 1e-4; I just mean, e.g., correlation across voxels should be > 0.9, and summary stat estimates should be within 10% of each other everywhere. Most of the time when algorithmic changes break something, they're not changing values at the 5th decimal, but throwing everything out of whack.

I would be perfectly happy with (a) 2 - 3 MATLAB-generated ALE images we can compare against our ALE implementation, and then (b) a test that compares our ALE results against our MKDADensity results (with tolerance adjusted as necessary). The other methods are less important, as they're not widely used and will probably be heavily refactored in the near future (certainly my plan is to do away with almost all of MKDAChi2 in favor of a stripped down version that re-uses much of the current CBMAEstimator code).

[tsalo]

@yifan0330 I've created a new repo, https://github.com/neurostuff/meta-analysis-comparison, where we can develop this idea further. We will probably also want to have some integration tests implemented in NiMARE, but I think that the new repo would be a good place to start working on this.

[yifan0330]

@tsalo Thanks for creating the new repo! I will get myself familiar with integration tests these days and work on this issue :)