cloudpathlib icon indicating copy to clipboard operation
cloudpathlib copied to clipboard

[WIP] Proof of concept fsspec integration

Open jayqi opened this issue 5 years ago • 2 comments

Implementation of FsspecClient and FsspecPath that work with an fsspec concrete filesystem implementation.

Follow up on #96


  • New abstract FsspecClient, FsspecPath classes. Each registered implementation in fsspec.registry has subclasses autogenerated, e.g., S3Path, GCSPath, HTTPPath.
  • FsspecPath will automatically dispatch to correct subclass, the same way CloudPath does. This dispatching is isolated to FsspecPath; CloudPath still only dispatches to original implementations.
    • Running environment variable FSSPEC_MODE=1 will replace CloudPath's dispatching so that it will dispatch to fsspec subclasses instead of the normal ones.
  • New extras.
    • [fsspec] will install fsspec.
    • Each extras from fsspec has been copied to [fsspec-<name>], e.g.fsspec-s3fs.
    • [fsspec-all] will install all fsspec extras.
    • original [all] does not include any fsspec extras

jayqi avatar Nov 16 '20 03:11 jayqi

🚀 Deployed on https://deploy-preview-109--gallant-agnesi-5f7bb3.netlify.app

github-actions[bot] avatar Nov 20 '20 06:11 github-actions[bot]

Codecov Report

Merging #109 (ba3d9f3) into master (8b230c3) will decrease coverage by 12.3%. The diff coverage is 15.7%.

@@            Coverage Diff            @@
##           master    #109      +/-   ##
=========================================
- Coverage    91.4%   79.1%   -12.4%     
=========================================
  Files           8      12       +4     
  Lines         680     792     +112     
=========================================
+ Hits          622     627       +5     
- Misses         58     165     +107     
Impacted Files Coverage Δ
cloudpathlib/fsspec/__init__.py 0.0% <0.0%> (ø)
cloudpathlib/fsspec/fsspecclient.py 0.0% <0.0%> (ø)
cloudpathlib/fsspec/fsspecpath.py 0.0% <0.0%> (ø)
cloudpathlib/fsspec/implementations.py 0.0% <0.0%> (ø)
cloudpathlib/cloudpath.py 89.8% <100.0%> (+0.1%) :arrow_up:

codecov[bot] avatar Nov 20 '20 06:11 codecov[bot]

See discussion in #96

pjbull avatar Jan 22 '23 02:01 pjbull