integration tests not warning about more voxels excluded from a mask

[handwerkerd]

Summary

As part of reviewing #358, I was looking at the resulting volumes. I noticed that a non-trivial number of voxels are masked out in #358 than in master. The discussion of why is in that PR, but the reason for this issue is that a change in results that is fairly dramatic seemed to pass the current tests in CI. I haven't yet looked into how the results are being compared in the integration tests, but it seems like some useful comparisons are missing.

Additional Detail

I locally ran the following command for both the master & PR358:

tedana -d /tmp/data/five-echo/p06.SBJ01_S09_Task11_e1.sm.nii.gz \
          /tmp/data/five-echo/p06.SBJ01_S09_Task11_e2.sm.nii.gz \
          /tmp/data/five-echo/p06.SBJ01_S09_Task11_e3.sm.nii.gz \
          /tmp/data/five-echo/p06.SBJ01_S09_Task11_e4.sm.nii.gz \
          /tmp/data/five-echo/p06.SBJ01_S09_Task11_e5.sm.nii.gz \
 -e 15.4 29.7 44.0 58.3 72.6 \
--out-dir tedana_out/PR358 \
--verbose 

Here is the resulting first volume of the optimally combined images with master on the left and PR358 on the right

Next Steps

• Have someone else replicate this comparison
• Do others agree this should have thrown up a failed CI test?
• Add code to test for this type of change
• Check to see if other non-trivial data changes aren't currently being examined.

[handwerkerd]

This relates to #59 which currently has a suggestion to consider nib-diff. From the Feb 2020 dev call, it sounds like we don't want CI to fail based on changes within files, but there was agreement that there should be an easier way for code reviewers to identify which files have internal changes and to relatively easily view those changes.

[tsalo]

What about outputting the difference images in the CI artifacts, so we could look at those directly and evaluate whether the change is too much more qualitatively?

[stale[bot]]

This issue has been automatically marked as stale because it has not had any activity in 90 days. It will be closed in 600 days if no further activity occurs. Thank you for your contributions to tedana:tada: !