bids-standard
Feature request: Documentation on multiple layers of derivatives
Unless I missed something in the documentation, I think it is currently unclear how pybids expect its users to deal with imbricated derivatives. The current tutorial explains how to access derivatives but without any reference to cases with multiple levels of derivatives. Take the following example: someone has a EEG dataset and wants to apply different modality of a two-level pipeline, lets say source reconstruction with two algorithms (lets say MNE and dSPM) following by connectivity analysis (lets say phase locking index (PLI)). The resulting BIDS directories would look something like:
- some_dataset/
- some_dataset/derivatives/
- some_dataset/derivatives/MNE/
- some_dataset/derivatives/MNE/derivatives/
- some_dataset/derivatives/MNE/derivatives/PLI
- some_dataset/derivatives/dSPM/
- some_dataset/derivatives/dSPM/derivatives/
- some_dataset/derivatives/dSPM/derivatives/PLI
It seems to me that the natural structure would be a tree structure of layouts, but what I've seen of PyBIDS code seems to consider derivatives as a flat list using a dictionary structure indexed by derivative names, such that for nested levels of derivatives need to be aware of the whole tree to avoid name clashing. In other words, it seems that we have to do something (conceptually) like
some_dataset_layout["dSPM-PLI"] instead of some_dataset_layout["dSPM"]["PLI"].
In any case, it is not very clear from the current documentation available how the users should deal with multiple levels of derivatives. If a section could be added to the current tutorial, or a second tutorial be made about working with derivatives with PyBIBS, I think it would be very useful to ensure a proper use of PyBIDS.
After more investigations, I guess that PYBIDS does structure its layout as a tree structure which would have to be accessed like this layout_object.derivatives['dSPM'].derivatives["PLI"]. I think that an example of that in the tutorial would be helpful for newcomers.
Yes @christian-oreilly, the derivatives are indexed recursively, so you would get the nested tree structure you found.
Feel free to open a PR to help out on this. We're definitely a bit behind on docs. Maybe a short "working with derivatives" section would be useful.
A useful part to know when working with derivatives, is that .get() accepts a scope argument which limits where files are indexed. So even though there is a recusive tree structure, you can search all of the files from the top level BIDSLayout
scope : str or list, optional
Scope of the search space. If passed, only
nodes/directories that match the specified scope will be
searched. Possible values include:
'all' (default): search all available directories.
'derivatives': search all derivatives directories.
'raw': search only BIDS-Raw directories.
'self': search only the directly called BIDSLayout.
<PipelineName>: the name of a BIDS-Derivatives pipeline.
