allow CuBIDS to concatenate subjects together from separate directories to be able to run with data on the S3

[audreymhoughton]

Use case example: pull subject directory into a temp directory run cubids-validate, cubids-group on that subject, do this with all subjects in an S3 bucket, and concatenate them into one group or validation file.

Let me know if I need to provide more clarity here.

[scovitz]

Putting this on our roadmap (would be a nice feature for future use)

[audreymhoughton]

@scovitz yes, please! I currently can't run cubids-group on any of my datasets, because they are on s3. I can only run cubids-validate.

[mattcieslak]

I think we could make a feature for cubids-add-nifti-info where the nifti files get downloaded from s3 temporarily and their info added to the sidecars. Then you'd only locally need the sidecars and empty files for the niftis to run cubids group

[audreymhoughton]

For some of our datasets, even having the sidecars locally is too much.

[mattcieslak]

In that case cubids probably couldn't fit all that metadata into memory at once. I think the best approach would be to split up huge datasets and do them in batches. I think this would be possible, and definitely easier than writing an s3 version of everything

[audreymhoughton]

Yeah, I think the key is to be able to concatenate multiple group files together. Then the method of running doesn't really matter as long as there's a cubids-group-merge function or something.