CPernet
[ENH] add system / smoke test
simple script to test the BIDS pipeline of a very simple dataset
- [x] improve parallel toolbox check comment
Currently this crashes
Reference to non-existent field 'anat_qa'.
Error in spmup_BIDS_QCtables (line 35)
Nsess(s) = length(subjects{s}.anat_qa);
Error in run_spmup_bids (line 193)
table_name = spmup_BIDS_QCtables(subjects, 'anat');
Error in moae (line 49)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
cannot use the MoAE dataset because slice timing is not defined. switching to the face repetition dataset.
got a bit further but now get this
Error using spm_vol>spm_vol_hdr (line 80)
File
"/home/remi/github/spmup/system_test/demos/raw/derivatives/sub-01/func/run1/rst_sub-01_task-facerepetition_rec-despiked_bold.nii"
does not exist.
Error in spm_vol (line 61)
v = spm_vol_hdr(deblank(P(i,:)));
Error in spmup_spatialcorr (line 44)
V = spm_vol(P);
Error in spmup_BIDS_preprocess (line 696)
[subject.func_qa{frun}.volume_outliers,
subject.func_qa{frun}.slice_outliers] = spmup_spatialcorr(realigned);
Error in run_spmup_bids (line 155)
[subject_sess,opt(s)] = spmup_BIDS_preprocess(subject_sess,
opt(s));
Error in moae (line 55)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
@CPernet Can you try to see if you can run this simple script on your side?
tried to fixed a few bugs on the way but just want to first make sure if this is just me.
why did it not generate /home/remi/github/spmup/system_test/demos/raw/derivatives/sub-01/func/run1/rst_sub-01_task-facerepetition_rec-despiked_bold.nii?
I did not try your script but ran the pipeline on ds0117 all the way with no error
Reference to non-existent field 'anat_qa'.
because no anat processed? this can be the case a dataset has no anat (rare but possible) we just need to add a if isfield statement, right?
Reference to non-existent field 'anat_qa'.because no anat processed? this can be the case a dataset has no anat (rare but possible) we just need to add a
if isfieldstatement, right?
did troubleshoot that one.
It seems that the approach for datasets with no session level folder was not optimal and triggered this.
Hence I did this to fix it:
if isempty(sess_names)
sess_names = '1';
sess_names = {''};
end
why did it not generate /home/remi/github/spmup/system_test/demos/raw/derivatives/sub-01/func/run1/rst_sub-01_task-facerepetition_rec-despiked_bold.nii?
no idea.
for ds000117
Error using fileparts
Too many input arguments.
Error in run_spmup_bids (line 122)
[~,sub_names] = fileparts(subjects{s}.func{session,:}); % e.g.
{'sub-53888_ses-02_acq-ep2d_task-faces_bold'} {'sub-53888_ses-02_acq-ep2d_task-rest_bold'}
Error in ds000117 (line 33)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
for ds000117
Error using fileparts Too many input arguments. Error in run_spmup_bids (line 122) [~,sub_names] = fileparts(subjects{s}.func{session,:}); % e.g. {'sub-53888_ses-02_acq-ep2d_task-faces_bold'} {'sub-53888_ses-02_acq-ep2d_task-rest_bold'} Error in ds000117 (line 33) [subjects, opt] = run_spmup_bids(BIDS, subjects, options);
Note that this does not seem to be a regression because of bids/run_spmup_bids.m
K>> fileparts(subjects{s}.func(session,:))
Error using fileparts (line 37)
Input must be a row vector of characters or string scalar.
37 error(message('MATLAB:fileparts:MustBeChar'));
your code
[~,sub_names] = fileparts(subjects{s}.func{session,:});
my code I have
[~,sub_names] = fileparts(subjects{s}.func(session,:));
I don't think your branch is up to date ?? see the difference you {} and me ()
subjects{s}.func(session,:)'
ans =
9×1 cell array
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run1/sub-01_ses-mri_task-facerecognition_run-01_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run2/sub-01_ses-mri_task-facerecognition_run-02_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run3/sub-01_ses-mri_task-facerecognition_run-03_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run4/sub-01_ses-mri_task-facerecognition_run-04_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run5/sub-01_ses-mri_task-facerecognition_run-05_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run6/sub-01_ses-mri_task-facerecognition_run-06_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run7/sub-01_ses-mri_task-facerecognition_run-07_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run8/sub-01_ses-mri_task-facerecognition_run-08_bold.nii'}
{'/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run9/sub-01_ses-mri_task-facerecognition_run-09_bold.nii'}
vs
subjects{s}.func{session,:}
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run1/sub-01_ses-mri_task-facerecognition_run-01_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run2/sub-01_ses-mri_task-facerecognition_run-02_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run3/sub-01_ses-mri_task-facerecognition_run-03_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run4/sub-01_ses-mri_task-facerecognition_run-04_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run5/sub-01_ses-mri_task-facerecognition_run-05_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run6/sub-01_ses-mri_task-facerecognition_run-06_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run7/sub-01_ses-mri_task-facerecognition_run-07_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run8/sub-01_ses-mri_task-facerecognition_run-08_bold.nii'
ans = '/data1/cpernet/ds000117/testing/sub-01/ses-mri/func/run9/sub-01_ses-mri_task-facerecognition_run-09_bold.nii'
which is why with your version it is not happy, with () it returns a cell array of names
PS: pushed the change for sess name empty
merged master branch and reverted the change from bids/run_spmup_bids.m
I get this.
Error using fileparts (line 37)
Input must be a row vector of characters or string scalar.
Error in run_spmup_bids (line 119)
[~,sub_names] = fileparts(subjects{s}.func(session,:)); % e.g.
{'sub-53888_ses-02_acq-ep2d_task-faces_bold'} {'sub-53888_ses-02_acq-ep2d_task-rest_bold'}
Error in ds000117 (line 33)
Note
Content of derivatives folder after unpacking.
When working with datalad on Windows I think you will want to get rid of those symlink because I remember it will be an issue.
Data must first be unlocked before copying: added a warning in bids matlad for those things.
https://github.com/bids-standard/bids-matlab/blob/255ff04f45d97c86665c9f82a377a6e9690637d0/%2Bbids/copy_to_derivative.m#L339
ds000117/derivatives/spmup
├── sub-01
│ ├── ses-meg
│ │ └── func
│ └── ses-mri
│ ├── anat
│ │ └── sub-01_ses-mri_acq-mprage_T1w.nii
│ ├── fieldmaps
│ │ ├── sub-01_ses-mri_magnitude1.nii -> ../../../.git/annex/objects/Q5/p9/MD5E-270688--dc7e47b55cf45cb84bd441c76ab079d6.nii/MD5E-s270688--dc7e47b55cf45cb84bd441c76ab079d6.nii
│ │ ├── sub-01_ses-mri_magnitude2.nii -> ../../../.git/annex/objects/pm/7w/MD5E-270688--13c81f5c0745d2a6a4c9f48dd0286d59.nii/MD5E-s270688--13c81f5c0745d2a6a4c9f48dd0286d59.nii
│ │ └── sub-01_ses-mri_phasediff.nii -> ../../../.git/annex/objects/kG/8k/MD5E-270688--0b8cddaf2117526602dd2d980e56a47a.nii/MD5E-s270688--0b8cddaf2117526602dd2d980e56a47a.nii
│ └── func
│ ├── run1
│ │ └── sub-01_ses-mri_task-facerecognition_run-01_bold.nii
│ ├── run2
│ │ └── sub-01_ses-mri_task-facerecognition_run-02_bold.nii
│ ├── run3
│ │ └── sub-01_ses-mri_task-facerecognition_run-03_bold.nii
│ ├── run4
│ │ └── sub-01_ses-mri_task-facerecognition_run-04_bold.nii
│ ├── run5
│ │ └── sub-01_ses-mri_task-facerecognition_run-05_bold.nii
│ ├── run6
│ │ └── sub-01_ses-mri_task-facerecognition_run-06_bold.nii
│ ├── run7
│ │ └── sub-01_ses-mri_task-facerecognition_run-07_bold.nii
│ ├── run8
│ │ └── sub-01_ses-mri_task-facerecognition_run-08_bold.nii
│ └── run9
│ └── sub-01_ses-mri_task-facerecognition_run-09_bold.nii
probably a change between 2018 and 2020+; I propose a
try
[~,sub_names] = fileparts(subjects{s}.func(session,:))
catch
for f=length(subjects{s}.func{session,:}):-1:1
[~,sub_names{f}] = fileparts(subjects{s}.func{session,f}))
end
end
try [~,sub_names] = fileparts(subjects{s}.func(session,:)) catch for f=length(subjects{s}.func{session,:}):-1:1 [~,sub_names{f}] = fileparts(subjects{s}.func{session,f})) end end
let me try
good you have an old matlab :-) or maybe not
what do you mean old? I updated from 2017 a few weeks ago. (Not a joke)
OK tried to hack my way around this.
But got there.
Error using run_spmup_bids (line 137)
somehow we cannot match BIDS_name with the spmup unpacked subject structure information, some silly matching error to
figure out - please reach out/raise an issue so we can fix it
Error in ds000117 (line 31)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
The problem:
K>> sub_names
sub_names =
9×1 cell array
{'sub-01_ses-mri_task-facerecognition_run-01_bold'}
{'sub-01_ses-mri_task-facerecognition_run-02_bold'}
{'sub-01_ses-mri_task-facerecognition_run-03_bold'}
{'sub-01_ses-mri_task-facerecognition_run-04_bold'}
{'sub-01_ses-mri_task-facerecognition_run-05_bold'}
{'sub-01_ses-mri_task-facerecognition_run-06_bold'}
{'sub-01_ses-mri_task-facerecognition_run-07_bold'}
{'sub-01_ses-mri_task-facerecognition_run-08_bold'}
{'sub-01_ses-mri_task-facerecognition_run-09_bold'}
K>> BIDS_name
BIDS_name =
'sub-01_ses-mri_task-facerecognition_run-02_bold.nii'
BIDS_name still has the .nii extension because run_ls has the .nii.gz one.
This is because BIDS contains the indexing by spm_BIDS of the raw dataset and not the one after unpacking.
run_ls
run_ls =
9×1 cell array
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-01_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-02_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-03_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-04_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-05_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-06_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-07_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-08_bold.nii.gz'}
{'/home/remi/github/spmup/system_test/ds000117/sub-01/ses-mri/func/sub-01_ses-mri_task-facerecognition_run-09_bold.nii.gz'}
we need to fix this line
maybe adding something about having a .nii here
if empty then loop and as you loop if .nii then do x?
(I don't understand how you got this error btw, I downloaded using datalad, then execute the code and no such error)
When working with datalad on Windows I think you will want to get rid of those symlink because I remember it will be an issue.
Data must first be unlocked before copying: added a warning in bids matlad for those things.
https://github.com/bids-standard/bids-matlab/blob/255ff04f45d97c86665c9f82a377a6e9690637d0/%2Bbids/copy_to_derivative.m#L339
Running into symlink issues even on linux
Copying data: /home/remi/github/spmup/system_test/ds000117/sub-03/ses-mri/fmap/sub-03_ses-mri_phasediff.nii
Error using spmup_BIDS_unpack>unzip_or_copy (line 425)
cp: not writing through dangling symlink '/home/remi/github/spmup/system_test/ds000117/derivatives/spmup/sub-03/ses-mri/fieldmaps/sub-03_ses-mri_phasediff.nii'
Error in spmup_BIDS_unpack (line 190)
parfor s=1:nb_sub
Error in ds000117 (line 29)
[BIDS, subjects] = spmup_BIDS_unpack(BIDS_dir, options);
I'm not saying all this is not useful, but really would that not be easier to focus on matlab 2020 and up rather than working on fixing issues for 5% of future users? unless the same applies to Octave
I'm not saying all this is not useful, but really would that not be easier to focus on matlab 2020 and up rather than working on fixing issues for 5% of future users? unless the same applies to Octave
yup that will defo affect octave.
see below
GNU Octave, version 7.1.0
Copyright (C) 1993-2022 The Octave Project Developers.
This is free software; see the source code for copying conditions.
There is ABSOLUTELY NO WARRANTY; not even for MERCHANTABILITY or
FITNESS FOR A PARTICULAR PURPOSE. For details, type 'warranty'.
Octave was configured for "x86_64-pc-linux-gnu".
Additional information about Octave is available at https://www.octave.org.
Please contribute if you find this software useful.
For more information, visit https://www.octave.org/get-involved.html
Read https://www.octave.org/bugs.html to learn how to submit bug reports.
For information about changes from previous versions, type 'news'.
>> fileparts({pwd})
error: fileparts: FILENAME must be a single string
error: called from
fileparts at line 42 column 5
Some functions still rely on cubehelix that was removed here: https://github.com/CPernet/spmup/commit/678c5aaa72df50cc616609e18fda487a1ff7bd88
Undefined function or variable 'cubehelix'.
Error in spmup_temporalSNR (line 240)
index = index+1; colormap(cubehelix(32,[1.15,0.1,4,1], [0,1], [0,0.85]))
Error in spmup_BIDS_preprocess (line 1235)
subject.func_qa{frun}.preproc_tSNR = spmup_temporalSNR(newname,subject.tissues);
Error in run_spmup_bids (line 162)
[subject_sess,opt(s)] = spmup_BIDS_preprocess(subject_sess, opt(s));
Error in ds000117 (line 31)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
Will switch this to use delimiter, '\t' which should be more backward compatible.
Error using readtable (line 197)
The file type 'delimitedtext' is not recognized.
Error in spmup_BIDS_1rstlevel (line 138)
events = readtable(subject.event{frun},'FileType','delimitedtext');
Error in run_spmup_bids (line 164)
subject_sess = spmup_BIDS_1rstlevel(subject_sess, opt(s));
Error in ds000117 (line 30)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
FYI octave does not have readtable or contains functions yet
OK... I think I am going to give up. Readtable cannot read TSV??!!!
Error using readtable (line 197)
The file extension '.tsv' is not recognized.
Error in spmup_BIDS_1rstlevel (line 138)
events = readtable(subject.event{frun},'Delimiter','\t');
Error in run_spmup_bids (line 164)
subject_sess = spmup_BIDS_1rstlevel(subject_sess, opt(s));
Error in ds000117 (line 30)
[subjects, opt] = run_spmup_bids(BIDS, subjects, options);
got a bit further but now get this
Error using spm_vol>spm_vol_hdr (line 80) File "/home/remi/github/spmup/system_test/demos/raw/derivatives/sub-01/func/run1/rst_sub-01_task-facerepetition_rec-despiked_bold.nii" does not exist. Error in spm_vol (line 61) v = spm_vol_hdr(deblank(P(i,:))); Error in spmup_spatialcorr (line 44) V = spm_vol(P); Error in spmup_BIDS_preprocess (line 696) [subject.func_qa{frun}.volume_outliers, subject.func_qa{frun}.slice_outliers] = spmup_spatialcorr(realigned); Error in run_spmup_bids (line 155) [subject_sess,opt(s)] = spmup_BIDS_preprocess(subject_sess, opt(s)); Error in moae (line 55) [subjects, opt] = run_spmup_bids(BIDS, subjects, options);
This gets reproduced in CI:
https://github.com/Remi-Gau/spmup/actions/runs/3348007893/jobs/5546607296#step:5:173
Error using readtable (line 197) The file extension '.tsv' is not recognized. Error in spmup_BIDS_1rstlevel (line 138) events = readtable(subject.event{frun},'Delimiter','\t'); Error in run_spmup_bids (line 164) subject_sess = spmup_BIDS_1rstlevel(subject_sess, opt(s)); Error in ds000117 (line 30) [subjects, opt] = run_spmup_bids(BIDS, subjects, options);
fixed