File access via fsspec

[chrisroat]

trafficstars

Feature Request

Problem

Accessing various filesystems (s3, gcsfs, azure, gdrive, dropbox) with the current implementation requires new modules for each system, providing the file-like methods/attributes.

Requirements

Switch file-storage backend to use fsspec.

Justification

This change would allow users to automatically use any fsspec-supported backend by loading the corresponding module, i.e. GCS vs gcsfs. Of course, the backend must support the operations needed by datajoint.

Alternative Considerations

As stated in the problem, a custom module is currently necessary for a new file system.

Related Errors

N/A

Screenshots

N/A

Additional Research and Context

The fsspec project is providing a common, pythonic interface across many kinds of local and remote filesystems. Several project have announced recently starting to use it -- for a major example, see the pandas 1.1 release.

[dimitri-yatsenko]

Excellent idea.