verified-sources icon indicating copy to clipboard operation
verified-sources copied to clipboard

support git filesystem in `filesystem` source

Open rudolfix opened this issue 1 year ago • 1 comments

Which source is this for adds git filesystem support to filesystem source

Describe the data you'd like, be specific, link the docs We want to support git in filesystem source the same way we support buckets. This will involve using (and possibly reimplementing) the filesystem provided by fsspec by default.

    • [ ] try if the default git filesystem implementation works, if not we'll need to implement our own. the problem I see is that last modification timestamp of the file in git repo is not returned by the default fsspec impl.
    • [ ] allow both local and remote git repositories. dlt has a good support for cloning and checking out the branches: https://github.com/dlt-hub/dlt/blob/master/dlt/common/git.py we also have support for giturlparse for remot repos
    • [ ] initially you can add all the code (including custom fsspec filesystem) to the filesystem verified source
    • [ ] if you need to modify library core, do it a fork from which we can install it for testing. after a review we'll merge it
    • [ ] please reproduce all the test cases by creating a repo with the test files used by other filesystems in filesystem source

rudolfix avatar Dec 12 '23 12:12 rudolfix