pybids icon indicating copy to clipboard operation
pybids copied to clipboard

Update indexer to treat .zarr as files not dirs

Open akhanf opened this issue 1 year ago • 1 comments

Microscopy file formats were recently updated in the standard to include ome-zarr, which are stored in a directory suffixed with .ome.zarr.

https://bids-specification.readthedocs.io/en/stable/modality-specific-files/microscopy.html#file-formats

This however doesn't work with pybids currently, since the pybids just indexes it as a directory.. This can cause issues if there is an ome.zarr folder with real data, since there are a large number of files in that folder which slow the indexing to a crawl..

This adds a quick and simple (albeit hardcoded) fix to ensure .zarr directories get indexed as files instead of directories. This enables us to get .ome.zarr microscopy files, and also prevents needless indexing of .zarr directories that contain a large number of files.

This solves my problem as I can now use ome.zarr in bids, seemingly without any negative side effects (?) - a more configurable solution would seem nicer of course, but not sure how often this would come up in the future (ie bids "files" that are actually directories), so not sure it would be worth the added config set-up?

Thoughts? Pinging @satra as someone in the BIDS community that was also involved in the ome-zarr standard.

Note: .ome.zarr does not yet seem to be in the pybids config, so using validate=False in the layout to test things..

akhanf avatar Mar 05 '24 01:03 akhanf

Codecov Report

All modified and coverable lines are covered by tests :white_check_mark:

Project coverage is 89.80%. Comparing base (6aaa5f2) to head (d556233).

Additional details and impacted files
@@           Coverage Diff           @@
##           master    #1046   +/-   ##
=======================================
  Coverage   89.79%   89.80%           
=======================================
  Files          63       63           
  Lines        7175     7178    +3     
  Branches     1372     1374    +2     
=======================================
+ Hits         6443     6446    +3     
  Misses        532      532           
  Partials      200      200           

:umbrella: View full report in Codecov by Sentry.
:loudspeaker: Have feedback on the report? Share it here.

codecov[bot] avatar Mar 05 '24 01:03 codecov[bot]

Agreed a more configurable solution would be nice. But I think that can wait until we depend on the BIDS schema, which does indicate extensions of data "files" that are actually directories.

effigies avatar May 05 '24 13:05 effigies