drivendataorg
Caching file/dir metadata for GSClient
This is an attempt to resolve #176 . It's a first attempt, I'd mainly like to know if this is what the maintainers had in mind. If it is, I can add some tests - which begs the question, what tests would we like to see?
Codecov Report
Merging #275 (10bb071) into master (b7f6010) will decrease coverage by
0.3%. The diff coverage is96.0%.
Additional details and impacted files
@@ Coverage Diff @@
## master #275 +/- ##
========================================
- Coverage 94.8% 94.4% -0.4%
========================================
Files 20 20
Lines 1314 1347 +33
========================================
+ Hits 1246 1272 +26
- Misses 68 75 +7
| Impacted Files | Coverage Δ | |
|---|---|---|
| cloudpathlib/local/implementations/gs.py | 96.9% <50.0%> (-3.1%) |
:arrow_down: |
| cloudpathlib/gs/gsclient.py | 93.7% <97.9%> (-1.0%) |
:arrow_down: |
| cloudpathlib/s3/s3client.py | 92.9% <0.0%> (-2.7%) |
:arrow_down: |
Thanks @Gilthans, nice work on a first pass on this. Definitely helpful for thinking about the problem.
Caching is a big enough feature/change that we'd prefer to spec it out before thinking about implementation. For example, are we confident we can implement it the same way across backends? What settings/config/env vars do we provide to end users to control behavior? Is this better implemented in the *Client or *Path objects? What exactly goes in the cache? Is the cache limited in size? Do we provide APIs for clearing the cache?
I think all of these are answerable, but worth making a plan ahead of time to make sure they don't catch us out later and increase the maintenance burden.
Sure, that does sound reasonable. How does such a discussion happen currently? Is it done strictly by the maintainer team, or is it an open discussion in an issue? I'd love to get the ball rolling since this feature would be very helpful in our project
@Gilthans We can do this collaboratively in an issue that is opened specifically for metadata caching. Thanks!