bids-specification icon indicating copy to clipboard operation
bids-specification copied to clipboard

dandischematools idea: "Filesystem" abstraction

Open yarikoptic opened this issue 1 year ago • 0 comments

Relates to the discussions about

  • "fake_paths" we had with @TheChymera [ref: TODO] needing to validate file path without having a file present on the file system (as checked e.g. by os.path.exists)
  • overlay of partial "copy" of the bids dataset on the local file system with e.g. fixed (meta)data files while the rest of the dataset residing on remote server , use case with @satra - https://dandiarchive.org/dandiset/000026/ (11TB in size, thus "tricky" to "just download it all")

I think it could start with validation code going through some LocalFilesystem class which would collect/implement all methods validator might potentially need (.lexists, .exists, .read_text, .load_json, .list_dir), and be able to provide such filesystem instance as an option to validate (and possibly other methods), or just implement .validate as its method. Then tools, like dandi-cli, could provide specific implementation, possibly mixing local + remote access (e.g. via fsspec) etc.

yarikoptic avatar Aug 03 '22 19:08 yarikoptic