datajoint
File access via fsspec
Feature Request
Problem
Accessing various filesystems (s3, gcsfs, azure, gdrive, dropbox) with the current implementation requires new modules for each system, providing the file-like methods/attributes.
Requirements
Switch file-storage backend to use fsspec.
Justification
This change would allow users to automatically use any fsspec-supported backend by loading the corresponding module, i.e. GCS vs gcsfs. Of course, the backend must support the operations needed by datajoint.
Alternative Considerations
As stated in the problem, a custom module is currently necessary for a new file system.
Related Errors
N/A
Screenshots
N/A
Additional Research and Context
The fsspec project is providing a common, pythonic interface across many kinds of local and remote filesystems. Several project have announced recently starting to use it -- for a major example, see the pandas 1.1 release.
