layout.add_derivatives requires 'PipelineDescription' not in BIDS specification 1.4.0

[pwrightkcl]

Using the add_derivatives method I get the following error: MANDATORY_DERIVATIVES_FIELDS['PipelineDescription.Name']) bids.exceptions.BIDSDerivativesValidationError: Every valid BIDS-derivatives dataset must have a PipelineDescription.Name field set inside 'dataset_description.json'.

I created my data_description.json based on the current BIDS specification and it does not mention the field PipelineDescription. It looks like the information is covered by 'GeneratedBy'. https://bids-specification.readthedocs.io/en/stable/03-modality-agnostic-files.html#derived-dataset-and-pipeline-description

I'm new to BIDS, so please forgive me if I'm missing it in the specs.

[tyarkoni]

Thanks for flagging this; looks like PyBIDS needs to be updated to reflect changes to the spec. Unfortunately this won't be completely trivial, as the ability to have multiple values in GeneratedBy will mess with the way derivatives are currently labeled internally.

I won't have any time to deal with this in the next ~2 weeks, so I suggest that for now you add a dummy value for 'PipelineDescription.Name' to your description file (i.e., Name nested within PipelineDescription rather than GeneratedBy), which should solve the problem.

[bpinsard]

I am having the same problem with fMRIPrep LTS output, and I think this can be confusing for end users. Should I hack all my datasets with a dummy value or is there any progress on fixing this issue? Should we modify fMRIPrep to have both fields in the dataset_description.json ?

[effigies]

This branch works. I haven't had time to finish it yet: https://github.com/bids-standard/pybids/pull/654

[seralouk]

This is really bug.

I just used fmriprep but I noticed that BIDSLayout would fail.

This is because the dataset_description.json file did not have a PipelineDescription key but instead it had GeneratedBy.

To make it work I had to manually change this file to:

[adelavega]

@effigies do you know if this is still a problem? Seems important if by default pybids can't open fmriprep outputs

[effigies]

Yes, it is.

[pwrightkcl]

I'm willing to help testing #654 if you need help with that, but I'll need telling what to test. I don't use fmriprep though, I'm generating my own BIDS filenames and JSON content.

[adelavega]

I'm a little confused: where did PipelineDescription.Name come from if its not in the spec? Was it just a proposal for BIDS-Derivatives that was over-written by GeneratedBy?

[effigies]

I'm willing to help testing #654 if you need help with that, but I'll need telling what to test. I don't use fmriprep though, I'm generating my own BIDS filenames and JSON content.

If you have an old-style derivative, you could test that:

1. The latest version of #654 will load it and warn you to upgrade the derivative.
2. Try upgrading the derivative.
3. Verify that the upgraded version works without warnings.

I'm a little confused: where did PipelineDescription.Name come from if its not in the spec? Was it just a proposal for BIDS-Derivatives that was over-written by GeneratedBy?

Exactly.

[effigies]

This should be resolved in the latest release. If not, let's open a new issue.