bids-specification
bids-specification copied to clipboard
dandischematools idea: "Filesystem" abstraction
Relates to the discussions about
- "fake_paths" we had with @TheChymera [ref: TODO] needing to validate file path without having a file present on the file system (as checked e.g. by
os.path.exists
) - overlay of partial "copy" of the bids dataset on the local file system with e.g. fixed (meta)data files while the rest of the dataset residing on remote server , use case with @satra - https://dandiarchive.org/dandiset/000026/ (11TB in size, thus "tricky" to "just download it all")
I think it could start with validation code going through some LocalFilesystem
class which would collect/implement all methods validator might potentially need (.lexists, .exists, .read_text, .load_json, .list_dir), and be able to provide such filesystem instance as an option to validate
(and possibly other methods), or just implement .validate
as its method. Then tools, like dandi-cli, could provide specific implementation, possibly mixing local + remote access (e.g. via fsspec) etc.