datamon Improvements to Key enumeration of store.go

Improvements to Key enumeration of store.go

Open kerneltime opened this issue 7 years ago • 4 comments

The localfs implementation needs to

[ ] More efficiently walk the space for localfs (faster is better)
[ ] The API should allow for pagination to walk an arbitrarily large number of files.
[ ] Support for prefix key enumeration

Nov 26 '18 23:11 kerneltime

ref: https://github.com/karrick/godirwalk

Jan 08 '19 13:01 kerneltime

GCS

[x] The API should allow for pagination to walk an arbitrarily large number of files.
[x] Support for prefix key enumeration
[x] Unit test

PR #68

Jan 21 '19 12:01 chandresh-pancholi

localfs case addressed by #404

We might still worry about the overall efficiency of key walking on a localfs. I don't see any better solution for now.

Feb 28 '20 11:02 fredbi

TODO: in core, replace calls to store.Keys by iterators. Plus: should deprecate Keys in storage.Store

Mar 06 '20 12:03 fredbi