criterion "Closest" is not in effect?

[yarikoptic]

Attn @pvelasco who might help to resolve ambiguity

@michael-sun 's dataset. We had to disable matching based on Shims, so the settings inthe heuristic are:

POPULATE_INTENDED_FOR_OPTS = {
    'matching_parameters': ['ImagingVolume', ],
    'criterion': 'Closest'
}

and then he had a scanning session like this:

(heudiconv-devel) [bids@rolando ses-03] > ls -1 /inbox/DICOM/202*/*/*/A003529/
001-anat-scout
002-anat-scout_MPR_sag
003-anat-scout_MPR_cor
004-anat-scout_MPR_tra
005-func_task-discorr_acq-ap
006-func_task-discorr_acq-pa
007-func_run-01_task-bodymap_acq-ap-mb8_SBRef
008-func_run-01_task-bodymap_acq-ap-mb8
009-func_run-01_task-bodymap_acq-ap-mb8_SBRef
010-func_run-01_task-bodymap_acq-ap-mb8
011-func_run-02_task-bodymap_acq-ap-mb8_SBRef
012-func_run-02_task-bodymap_acq-ap-mb8
013-func_run-03_task-bodymap_acq-ap-mb8_SBRef
014-func_run-03_task-bodymap_acq-ap-mb8
015-func_run-04_task-bodymap_acq-ap-mb8_SBRef
016-func_run-04_task-bodymap_acq-ap-mb8
017-func_run-06_task-bodymap_acq-ap-mb8_SBRef
018-func_run-06_task-bodymap_acq-ap-mb8
019-func_run-07_task-bodymap_acq-ap-mb8_SBRef
020-func_run-07_task-bodymap_acq-ap-mb8
021-func_run-08_task-bodymap_acq-ap-mb8_SBRef
022-func_run-08_task-bodymap_acq-ap-mb8
023-func_run-08_task-bodymap_acq-ap-mb8_SBRef
024-func_run-08_task-bodymap_acq-ap-mb8
025-anat-scout
026-anat-scout_MPR_sag
027-anat-scout_MPR_cor
028-anat-scout_MPR_tra
029-func_task-discorr_acq-ap
030-func_task-discorr_acq-pa
031-func_run-09_task-pinel_localizer_acq-ap-mb8_SBRef
032-func_run-09_task-pinel_localizer_acq-ap-mb8
033-anat-T1w_acq-MPRAGE-p3-08mm-1
034-anat-T1w_acq-MPRAGE-p3-08mm-2

so that run-09 should have gone to the 2nd set of _acq-ap/pa's in fieldmap.

full reproin-based heuristic which just remaps into reproin

# Provide mapping into reproin heuristic names

from heudiconv.heuristics import reproin

from heudiconv.heuristics.reproin import *

POPULATE_INTENDED_FOR_OPTS = {
    'matching_parameters': ['ImagingVolume', ],
    'criterion': 'Closest'
}

protocols2fix.update({
    '':  # for any study given.  Needs recent heudiconv
        [   
            ('AAHead_Scout_.*', 'anat-scout'),
            # add _run-01 for consistentcy with later ones
            ('T1w_acq', 'T1w_run-01_acq'),
            ('AAHead_Scout_.*', 'anat-scout'),
            (r'task-bodymap_',  r'task-MUSTNOTHAPPEN_'),
            ('task-bodymapST1',  r'task-bodymapST1'),
            ('task-bodymapST2',  r'task-bodymapST2'),
            ('task-pinel.localizer',  r'task-pinellocalizer'),

            # also the same as above...
            # problematic case -- multiple identically named pepolar fieldmap runs
            # I guess we will just sacrifice ability to detect canceled runs here.
            # And we cannot just use _run+ since it would increment indepdently
            # for ap and then for pa.  We will rely on having ap preceding pa.
            # Added  _acq-mb8  so they match the one in funcs
            # yoh: no need since only a single one AFAIK
            ('func_task-discorr_acq-ap', r'fmap-epi_dir-ap_acq-mb8'),
            ('func_task-discorr_acq-pa', r'fmap-epi_dir-pa_acq-mb8'),
            # but removing -ap in funcs so we do not introduce non-matching _acq
            ('_acq-ap-mb8', '_acq-mb8'),

        ],
})

# We need to overload to be able to feed scans from varying
# accessions as for ses02 tasks being scanned in ses04.
# Fix to reproin sent in https://github.com/nipy/heudiconv/pull/508
def fix_canceled_runs(seqinfo):
    return seqinfo
reproin.fix_canceled_runs = fix_canceled_runs

but as a result we got all functionals assigned to the first fieldmap sessions

(heudiconv-devel) [bids@rolando ses-03] > grep -A13 -e IntendedFor fmap/sub-SID001651_ses-03_acq-mb8_dir-ap_epi__dup-01.json
  "IntendedFor": [
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_bold__dup-01.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref__dup-01.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-02_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-03_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-04_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-06_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-07_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_bold.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_bold__dup-01.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref__dup-01.nii.gz",
    "ses-03/func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_bold.nii.gz"
],

and no IntendedFor in the 2nd:

(heudiconv-devel) [bids@rolando ses-03] > grep -A13 -e IntendedFor fmap/sub-SID001651_ses-03_acq-mb8_dir-ap_epi.json 
(heudiconv-devel) [bids@rolando ses-03] > 

and all geometries seems to be matching

(heudiconv-devel) [bids@rolando ses-03] > nib-ls fmap/sub-SID001651_ses-03_acq-mb8_dir-*nii.gz func/*nii.gz
fmap/sub-SID001651_ses-03_acq-mb8_dir-ap_epi__dup-01.nii.gz                  uint16 [ 82,  82,  56,   2]  2.68x2.68x2.70x7.22   sform
fmap/sub-SID001651_ses-03_acq-mb8_dir-ap_epi.nii.gz                          uint16 [ 82,  82,  56,   2]  2.68x2.68x2.70x7.22   sform
fmap/sub-SID001651_ses-03_acq-mb8_dir-pa_epi__dup-01.nii.gz                  uint16 [ 82,  82,  56,   2]  2.68x2.68x2.70x7.22   sform
fmap/sub-SID001651_ses-03_acq-mb8_dir-pa_epi.nii.gz                          uint16 [ 82,  82,  56,   2]  2.68x2.68x2.70x7.22   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_bold__dup-01.nii.gz   int16 [ 82,  82,  56,   2]  2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_bold.nii.gz           int16 [ 82,  82,  56, 1185] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref__dup-01.nii.gz  int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-02_bold.nii.gz           int16 [ 82,  82,  56, 1183] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-02_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-03_bold.nii.gz           int16 [ 82,  82,  56, 1193] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-03_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-04_bold.nii.gz           int16 [ 82,  82,  56, 1225] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-04_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-06_bold.nii.gz           int16 [ 82,  82,  56, 1225] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-06_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-07_bold.nii.gz           int16 [ 82,  82,  56, 1225] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-07_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_bold__dup-01.nii.gz   int16 [ 82,  82,  56, 1169] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_bold.nii.gz           int16 [ 82,  82,  56, 1194] 2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref__dup-01.nii.gz  int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref.nii.gz          int16 [ 82,  82,  56]       2.68x2.68x2.70        sform
func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_bold.nii.gz      int16 [ 82,  82,  56, 692]  2.68x2.68x2.70x0.46   sform
func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_sbref.nii.gz     int16 [ 82,  82,  56]       2.68x2.68x2.70        sform

[pvelasco]

@yarikoptic , I can take a look. Where is the dataset?

So the issue is that the acquisition time for run 009 is closer to run 029 than to run 005, but it is still assigned to run 005, right? (I mean, all of them are assigned to 005...)

[yarikoptic]

Unfortunately it isn't public but I might be able to share jsons later if needed. And yes, your summary of the problem is matching my understanding

[pvelasco]

I would need to find out, for runs 005, 009 and 029, the full sform (including the 4th column) and the acquisition times. The 'ImagingVolume' criterium checks not only the voxel size and image dimensions and orientation, but also the position...

[Michael-Sun]

Hi @pvelasco thanks for helping us out. I'm attaching the relevant .json files from the BIDS folder in question. Not sure if this is all you need, but let me know if this helps and if there's anything else I should include. sub-SID001651_ses-03.zip

[pvelasco]

Hi @Michael-Sun,

Inspecting the json files you provided:

cd ses-03
for key in '"ImageOrientationPatient"' '"ImagePositionPatient"'; do
  grep "$key" fmap/*_dir-pa*.json func/*.json
  echo "-------------"
done

results in:

fmap/sub-SID001651_ses-03_acq-mb8_dir-pa_epi.json:      "ImageOrientationPatient": [0.99820822828092, -0.0542894215139, 0.02515932638135, 0.054410522673, 0.99851001148662, -0.0041535507038],
fmap/sub-SID001651_ses-03_acq-mb8_dir-pa_epi__dup-01.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-01.json:      "ImageOrientationPatient": [0.99980001903975, -0.0060201015159, -0.0190704039247, 0.00785253383033, 0.99520260152069, 0.09751984228293],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-02.json:      "ImageOrientationPatient": [0.99980001903975, -0.0060201015159, -0.0190704039247, 0.00785253383033, 0.99520260152069, 0.09751984228293],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_sbref__dup-01.json:      "ImageOrientationPatient": [0.99980001903975, -0.0060201015159, -0.0190704039247, 0.00785253383033, 0.99520260152069, 0.09751984228293],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_sbref__dup-02.json:      "ImageOrientationPatient": [0.99980001903975, -0.0060201015159, -0.0190704039247, 0.00785253383033, 0.99520260152069, 0.09751984228293],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_bold__dup-01.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref__dup-01.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-02_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-03_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-04_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-06_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-07_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref__dup-01.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_bold.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_sbref.json:      "ImageOrientationPatient": [0.99743361764464, 0.01600007239661, 0.06978664682901, -0.0125347994478, 0.99867987365345, -0.0498135399499],
-------------
fmap/sub-SID001651_ses-03_acq-mb8_dir-pa_epi.json:      "ImagePositionPatient": [-930.13969629272, -856.31611461824, -87.546896391471],
fmap/sub-SID001651_ses-03_acq-mb8_dir-pa_epi__dup-01.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-01.json:      "ImagePositionPatient": [-891.64205832792, -875.85452927781, -124.58864994879],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-02.json:      "ImagePositionPatient": [-891.64205832792, -875.85452927781, -124.58864994879],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_sbref__dup-01.json:      "ImagePositionPatient": [-891.64205832792, -875.85452927781, -124.58864994879],
func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_sbref__dup-02.json:      "ImagePositionPatient": [-891.64205832792, -875.85452927781, -124.58864994879],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_bold__dup-01.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-01_sbref__dup-01.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-02_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-03_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-04_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-06_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-07_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-bodymapST_acq-mb8_run-08_sbref__dup-01.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_bold.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
func/sub-SID001651_ses-03_task-pinellocalizer_acq-mb8_run-09_sbref.json:      "ImagePositionPatient": [-871.74966266113, -917.94381402895, -70.845461364756],
-------------

The "ImagePositionPatient" and "ImageOrientationPatient" of fmap/sub-SID001651_ses-03_acq-mb8_dir-<pa|ap>_epi__dup-01.json match those of all of the func runs except for func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-01.json and func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-02.json (and their respective _sbrefs). And the "ImagePositionPatient" and "ImageOrientationPatient" of fmap/sub-SID001651_ses-03_acq-mb8_dir-<pa|ap>_epi.json don't match any func run, so they no run is assigned to them.

So heudiconv assigns all of those functional runs to fmap/sub-SID001651_ses-03_acq-mb8_dir-<pa|ap>_epi__dup-01.json, and func/sub-SID001567_ses-03_task-bodymapST1_acq-mb8_run-01_bold__dup-0<1|2>.json doesn't get assigned to any fmap because the "ImagePositionPatient" and "ImageOrientationPatient" are different.