WIP: [ENH] Pyafq visualization of the recognized bundles for reportlets

[36000]

Closes #421

[36000]

@mattcieslak you should be able to see in the diffs where I can get the path to the visualization, not sure how to move it to reportlets dir yet.

[36000]

Now also addresses #434

[36000]

@smeisler current thinking is there's a bug in how pyAFQ imports qsiprep mapping, quickest fix is to tell pyAFQ to make its own mapping using SyN. you can use this PR or change this line in the code: mapping_definition=itk_map, to: mapping_definition=None, # try in-house mapping

Let me know if this fixes the problem.

[smeisler]

I tried changing the mapping_definition to None, but I am still left with a very sparse tractogram (Original 10M streamlines, tcksift to 2.5M streamlines, AFQ version has 3900 streamlines). As a result, some tract segmentations (notably bilateral arcuate) are practically empty. Is this a problem with how AFQ cleans streamlines? This wasn't with QSIprep outputs, though. This is using HCP-preprocessed data, with most recent PyAFQ version:

import os
from AFQ.api.participant import ParticipantAFQ
from AFQ.definitions.image import ImageFile
from AFQ.definitions.mapping import ItkMap

# Get paths to data
dwi_path = os.path.join(work_dir,'raw','data.nii.gz') # Preprocessed DWI
mask_path = os.path.join(work_dir,'raw','nodif_brain_mask.nii.gz') # Brain Mask
brain_mask_definition = ImageFile(path=mask_path)
# Gradient info (output from MRtrix dwigradcheck)
bvec_path = os.path.join(work_dir,'raw','bvecs_corr')
bval_path = os.path.join(work_dir,'raw','bvals_corr')
# SIFTed tractogram
tck_path = os.path.join(work_dir,'fodf','sifttracks.tck')

print('RUNNING AFQ')
myafq = ParticipantAFQ(
            dwi_path, bval_path, bvec_path, output_dir,
            import_tract=tck_path,
            brain_mask_definition=brain_mask_definition,
            mapping_definition=None)
myafq.export_all()

[36000]

Do you have the output that pyAFQ prints to the command line?

[smeisler]

Below. No apparent errors.

WARNING:py.warnings:The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
WARNING:py.warnings:The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
RUNNING AFQ
100%|████████████████████████████████| 936256/936256.0 [17:53<00:00, 872.28it/s]
Optimizing level 2 [max iter: 10000]
Optimizing level 1 [max iter: 1000]
Optimizing level 0 [max iter: 100]
Optimizing level 2 [max iter: 10000]
Optimizing level 1 [max iter: 1000]
Optimizing level 0 [max iter: 100]
Optimizing level 2 [max iter: 10000]
Optimizing level 1 [max iter: 1000]
Optimizing level 0 [max iter: 100]
WARNING:AFQ.api.seg:0 invalid streamlines removed
  0%|                                    | 32/401313 [00:03<13:27:13,  8.29it/s]WARNING:py.warnings:The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
WARNING:py.warnings:The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
..... (REPEATS THIS WARNING ALOT)
WARNING:py.warnings:The ability to pass arguments to BIDSLayout that control indexing is likely to be removed in future; possibly as early as PyBIDS 0.14. This includes the `config_filename`, `ignore`, `force_index`, and `index_metadata` arguments. The recommended usage pattern is to initialize a new BIDSLayoutIndexer with these arguments, and pass it to the BIDSLayout via the `indexer` argument.
100%|████████████████████████████████| 401313/401313 [00:37<00:00, 10653.19it/s]
100%|████████████████████████████████| 343966/343966 [00:19<00:00, 17994.62it/s]
100%|████████████████████████████████| 147067/147067 [00:09<00:00, 15534.53it/s]
100%|████████████████████████████████| 173583/173583 [00:11<00:00, 15563.62it/s]
100%|████████████████████████████████| 184135/184135 [00:11<00:00, 16063.29it/s]
100%|████████████████████████████████| 169073/169073 [00:09<00:00, 16907.63it/s]
100%|████████████████████████████████| 443116/443116 [00:32<00:00, 13648.43it/s]
100%|████████████████████████████████| 399887/399887 [00:30<00:00, 13036.09it/s]
100%|█████████████████████████████████| 366762/366762 [00:50<00:00, 7256.07it/s]
100%|█████████████████████████████████| 363001/363001 [00:50<00:00, 7180.47it/s]
100%|█████████████████████████████████| 519649/519649 [00:59<00:00, 8662.68it/s]
100%|█████████████████████████████████| 427696/427696 [00:59<00:00, 7188.30it/s]
100%|█████████████████████████████████| 317261/317261 [00:32<00:00, 9672.97it/s]
100%|█████████████████████████████████| 229996/229996 [00:28<00:00, 8192.01it/s]
100%|████████████████████████████████| 190143/190143 [00:13<00:00, 14018.79it/s]
100%|████████████████████████████████| 104629/104629 [00:06<00:00, 15266.17it/s]
100%|█████████████████████████████████| 254307/254307 [00:36<00:00, 6928.22it/s]
100%|████████████████████████████████| 386353/386353 [00:31<00:00, 12407.06it/s]
WARNING:AFQ.api:To make plots in pyAFQ, you will need to install the relevant plotting software packages.You can do that by installing pyAFQ with `pip install AFQ[plot]`

[36000]

@smeisler would it be possible to share the derivatives of the original run through qsiprep ? For the run when you are using pyAFQ directly, make sure you are pointing it to the DWI data that is preprocessed with qsiprep.

[smeisler]

The data for one subject may be found here: https://drive.google.com/drive/folders/1j6QUBaO2lAYNUhSsWLzlYyUAB8aCLIDF?usp=sharing

For the result I described in my last message, I was sure that I input the HCP preprocessed DWI image, with gradient information confirmed by dwigradcheck

edited to add the data are currently being uploaded there and should be done within 10 minutes.

[36000]

@smeisler just to keep you updated, I downloaded the data and tried a variety of tractography and mapping methods on it. I think everything works fine except none of them can find the arcuate with more than ~10 streamlines (ie, I don't think the 2.5 million streamlines to 4000 streamlines in itself is a problem, as long as the bundles are found, which the arcuate is not). Anyways, I am not sure why we are having trouble finding the arcuate in this dataset, but I am tinkering with the bundle recognition algorithm now and will let you know when I can get it to work.

[smeisler]

Thanks, I appreciate the update!

[mattcieslak]

@smeisler you're using HCP minimally preprocessed data as the input? there may be some issues with this because they write LAS+ results which can cause issues with how QSIPrep interprets the bvecs.

[smeisler]

To clarify, this is more of a PyAFQ issue than it is QSIPrep, but I have had this issue of sparse tractograms and segmentations with both QSIPrep processed data (google drive above) and HCP preprocessed data. For the HCP preprocessed data, I made tractograms with analagous commands to the QSIPrep MRtrix workflows which worked fine.

[36000]

pyAFQ may be more sensitive to this issue than other pipelines. @mattcieslak is there a way to validate that this is not an issue, for example by looking at the tractography or DWI files?

Edit: it would have to be a subtle issue because most of the bundles look credible.

[mattcieslak]

@smeisler are you using the pyAFQ recon workflow in qsiprep on your HCP data? Or standalone PyAFQ? If it's through the recon workflow there may be a y-flip in the bvecs. Would you two be up for a quick zoom meeting tomorrow to look into this?

[smeisler]

I was using PyAFQ standalone. I would be free tomorrow for a zoom meeting.

[36000]

I am free tomorrow before 10 am PST, between 11:45 and 12:45 PST, and after 6:30 pm PST

[mattcieslak]

I completely dropped the ball on this. Does anyone remember if this is resolved?

[smeisler]

I believe it was resolved