[FEATURE REQUEST] Saving Eddy-outputs in output folder for use in EddyQC

[ajschadler12]

In order to use FSL's EddyQC with data preprocessed using qsiprep, I need to save the working directory locally using -w WORK_DIR.

Is there an option to save the necessary eddy files (e.g. index.txt, acqparams.txt, etc.) to a folder in a subject's output directory?

[rwmair]

I'll second this request. The quantitative information from the EddyQC report is very useful. Having all the eddy inputs/outputs saved - or even having the option for qsiprep to run EddyQC - would be awesome.

I've been doing the same thing this week - copying over the working directory to my local space, but I still can't run EddyQC as the *.eddy_movement_rms file is missing, and the *.eddy_parameters file doesn't seem to suffice. Here's the error message:

File "/ncf/nrg/sw/apps/fsl/6.0.4/fslpython/envs/fslpython/lib/python3.7/site-packages/eddy_qc/QUAD/quad.py", line 353, in main raise ValueError('Motion estimates and/or eddy estimated parameters are missing!') ValueError: Motion estimates and/or eddy estimated parameters are missing!

Is there a parameter in the eddy_params.json that would give me more of the outputs in the working directory? Not all those eddy_params.json parameters seem functional - eg: I set "residuals": True, - but still don't get a residuals file output. I'm still wrestling with getting eddy_cuda to run within qsiprep on our gpu system to see if the resultant *.eddy_movement_over_time file will give enough info for eddyQC to run.

Thanks

[mattcieslak]

Which directory are you using from the working directory?

All the motion information is available in the confounds.tsv file, but it may need to be reformatted to match what eddyqc is expecting.

[ajschadler12]

Which directory are you using from the working directory?

I am pulling eddy outputs from a few folders in <subj>/<dwi_#_preproc>/hmc_sdc_wf

gather_inputs/

• eddy_index.txt
• eddy_acqp.txt

topup/

• fieldmap_HZ.nii.gz

eddy/

• eddy_corrected* (all relevant files)

[rwmair]

Which directory are you using from the working directory?

All the motion information is available in the confounds.tsv file, but it may need to be reformatted to match what eddyqc is expecting.

Hi Matt, thanks for this great tool!

I'll try re-making the *.eddy_movement_rms file with the info from the confounds.tsv file and see if that lets eddyQC run.

AJ - I found all those files used as the inputs to eddy in qsiprep, but /<dwi_#_preproc>/hmc_sdc_wf/eddy only has these files from eddy output:

command.txt eddy_corrected.eddy_cnr_maps.nii.gz eddy_corrected.eddy_outlier_n_sqr_stdev_map eddy_corrected.eddy_parameters eddy_corrected.eddy_rotated_bvecs eddy_corrected.nii.gz

When I run eddy manually, I get a total of 18 files output from each eddy command (admittedly with s2v which I'm not using thru qsiprep), but haven't been able to find others in the qsiprep directory structure. I guess having gathered the inputs for eddy used by qsiprep, I could run eddy again outside qsiprep, although that seems a painful way to go for a large study. Thanks!

[mattcieslak]

since qsiprep uses nipype, any files produced in the working directory that aren't used later in the pipeline get deleted. So it's possible that eddy writes a bunch of files that get cleaned up at the end of the qsiprep run. You can turn this off by using a nipype config file (be sure it's visible in the container) and setting remove_unnecessary_outputs to false. More info is available here. If you can confirm which files eddyqc needs I can make sure those get added to the outputs directory.

At the moment QSIPrep produces files that can be used by an interactive QC tool. I haven't documented this yet, but you can use qsiprep_group_report to generate a group QC file and you can view it using the dmriprep viewer. There is an unlisted youtube walkthrough that shows how to use this. Just replace the "expert_qc" directory with your qsiprep output directory after running qsiprep_group_report. You only need the dwiqc.json files and directory structure on your local machine.

If you'd be willing to write up a tutorial on this, I'd be happy to help you get started over a zoom call or something.

[rwmair]

Matt, that's an awesome viewer/interactive tool. Will look into that shortly.

For eddyQC, i've been running it with the -v flag now, and it seems to be complaining about the lack of the eddy_corrected.eddy_values_of_all_input_parameters file. This is just a listing of all eddy parameters used in the job, which is the same information you provide in <scan>/hmc_sdc_wf/eddy/_report/report.rst and in the .json file in <scan>/hmc_sdc_wf/eddy but i haven't been able to quickly transform one of those into a format eddyQC would accept. The remove_unneccesary_outputs option in a config file might be the way to go.

I'd definitely keep the following: eddy_corrected.eddy_values_of_all_input_parameters eddy_corrected.eddy_movement_rms eddy_corrected.eddy_outlier_map eddy_corrected.eddy_outlier_n_stdev_map eddy_corrected.eddy_outlier_report eddy_corrected.eddy_restricted_movement_rms eddy_corrected.eddy_movement_over_time (the last one if s2v correction is run with eddy_cuda)

Is it possible for the user to configure that?

Thanks

[ajschadler12]

since qsiprep uses nipype, any files produced in the working directory that aren't used later in the pipeline get deleted. So it's possible that eddy writes a bunch of files that get cleaned up at the end of the qsiprep run. You can turn this off by using a nipype config file (be sure it's visible in the container) and setting remove_unnecessary_outputs to false. More info is available here. If you can confirm which files eddyqc needs I can make sure those get added to the outputs directory.

At the moment QSIPrep produces files that can be used by an interactive QC tool. I haven't documented this yet, but you can use qsiprep_group_report to generate a group QC file and you can view it using the dmriprep viewer. There is an unlisted youtube walkthrough that shows how to use this. Just replace the "expert_qc" directory with your qsiprep output directory after running qsiprep_group_report. You only need the dwiqc.json files and directory structure on your local machine.

If you'd be willing to write up a tutorial on this, I'd be happy to help you get started over a zoom call or something.

I'd be happy to take a look into this. If you wanted to meet up via Zoom, anytime next week can work for me!

[rwmair]

since qsiprep uses nipype, any files produced in the working directory that aren't used later in the pipeline get deleted. So it's possible that eddy writes a bunch of files that get cleaned up at the end of the qsiprep run. You can turn this off by using a nipype config file (be sure it's visible in the container) and setting remove_unnecessary_outputs to false. More info is available here. If you can confirm which files eddyqc needs I can make sure those get added to the outputs directory.

Thanks Matt. The nipype.cfg file made all the difference. That nicely retained all the eddy output files (plus topup output files, which might also be useful at some point). EddyQC ran without any issues on the qsiprep output after that. This workaround is good for me, but if all the eddy_correct.* output files could be retained in a future release of qsiprep, that would be even better.

[julfou81]

Hi @mattcieslak, I just tried qsiprep for the first time, it is a great tool! Thank you for providing it to the community. I would be interested to try the qsiprep_group_report functionality. How do you launch this command? Is it possible to use it via a singularity image of qsiprep? Sorry if it is a silly question!

[CaitlinLloyd]

Hi @mattcieslak, thanks again for this great pipeline! I am struggling a little bit to produce the eddy files, and wondered what I might be doing wrong? We have a nipype file that sets remove_unnecessary_outputs to false, and placed in the root directory, but still cannot produce the eddy files. Do you have any thoughts? Is there an additional argument we should be using to run qsiprep? The --config flag does not seem to be recognised.

[rwmair]

Hi @CaitlinLloyd, the nippy.cfg file with remove_unnecessary_outputs = false retains all the unnecessary outputs in the working directory where qsiprep is running. But depending on your computing architecture (eg HPC cluster with job sumibbsion) this may get automatically deleted when the qsiprep job finishes. I had to string together commands to copy the entire working directory from the scratch space on our cluster back to my local directory, after which I could copy out the eddy output files. This working directory is 5-10GB depending on the size of your dataset - which I guess is why qsiprep by default doesn't retain all this stuff. So you may have to watch your own storage quota if running a lot of them.

[rwmair]

I should have added - once you've saved the working directory, you'll find the eddy output in: <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/eddy In addition to that entire folder I copy these additional files from the working directory to run eddy_quad: <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/gather_inputs/eddy*.txt <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/pre_eddy_b0_ref_wf/enhance_and_mask_b0/topup_imain_corrected_avg_mask.nii.gz <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/topup_to_eddy_reg/topup_reg_image_flirt.mat <workdir_name>/qsiprep_wf/single_subject_<subjID>_wf/dwi_preproc_<scan_ID>_wf/hmc_sdc_wf/topup/fieldmap_HZ.nii.gz as well as the bvals from qsiprep output and the slspec file from a common location.

[CaitlinLloyd]

Thank you for this! I am still not seeing anything other than these files in the eddy subfolder within qsiprep workingdir: _inputs.pklz _node.pklz _report command.txt eddy_corrected.eddy_command_txt eddy_corrected.eddy_post_eddy_shell_PE_translation_parameters eddy_corrected.eddy_values_of_all_input_parameters

As far as I can tell nothing is being removed automatically. I think maybe we have the cfg file set up incorrectly or not visible to the container somehow? Would you mind sharing an example of the nipype.cfg file, just so I can rule that out?

[mattcieslak]

Are you running this via docker or singularity? I wonder if the nipype config isn't visible inside the container

[CaitlinLloyd]

docker, here is command in case you see anything odd!

nohup /usr/bin/docker service create --constraint 'node.hostname==gpu' --replicas 1 --reserve-cpu 8 --reserve-memory 16g --mode replicated --restart-condition none --name=qsi_id -un$qpath --mount type=bind,source=$indir,destination=/input,readonly=true --mount type=bind,source=$outdir,destination=/output --mount type=bind,source=$workdir,destination=/workdir --mount type=bind,source=/DATA /bin/freesurferlicense.txt,destination=/opt/freesurfer/license.txt,readonly=true --mount type=bind,source=/DATA/testing/nipype.cfg,destination=/home/root/.nipype/nipype.cfg pennbbl/qsiprep /input /output participant -w /workdir $addarg > $log 2>&1 &

[mattcieslak]

I've actually never seen a docker service being used for qsiprep. What happens if you mount the config file at /root/.nipype/nipype.cfg?

[CaitlinLloyd]

Sadly that does not work either . We have tried a few different locations without too much luck, but not quite sure where to go next!

[mattcieslak]

It looks like you can set where nipype will look for the config file with the NIPYPE_CONFIG_DIR environment variable. If you set this in your container and mount the config file at ${NIPYPE_CONFIG_DIR}/nipype.cfg then it should work regardless of which user you're running qsiprep as

[rwmair]

It must be that your qsiprep job is not seeing the nipype config file. Here's what the eddy sub-folder in the working directory contained on one of my recent runs: _0x13da34fc6dfdee0ae211ccc7f4259587.json command.txt eddy_corrected.eddy_cnr_maps.nii.gz eddy_corrected.eddy_command_txt eddy_corrected.eddy_mbs_first_order_fields.nii.gz eddy_corrected.eddy_movement_over_time eddy_corrected.eddy_movement_rms eddy_corrected.eddy_outlier_free_data.nii.gz eddy_corrected.eddy_outlier_map eddy_corrected.eddy_outlier_n_sqr_stdev_map eddy_corrected.eddy_outlier_n_stdev_map eddy_corrected.eddy_outlier_report eddy_corrected.eddy_parameters eddy_corrected.eddy_post_eddy_shell_alignment_parameters eddy_corrected.eddy_post_eddy_shell_PE_translation_parameters eddy_corrected.eddy_residuals.nii.gz eddy_corrected.eddy_restricted_movement_rms eddy_corrected.eddy_rotated_bvecs eddy_corrected.eddy_values_of_all_input_parameters eddy_corrected.nii.gz _inputs.pklz _node.pklz _report result_eddy.pklz

I am running eddy with just about everything - cuda, slice-to-vol correction with mporder=8, cnr maps and residuals on, movement-by-susceptability correction, so that adds to the plethora of files. Your eddy job may not be generating all of these.