cloudpathlib icon indicating copy to clipboard operation
cloudpathlib copied to clipboard

Iterative walk for sparsely traversing large nested directories

Open clane9 opened this issue 7 months ago • 1 comments

I'm using CloudPath.walk to search a large nested directory, but I'm only descending into a very small subset of subdirectories (by removing elements from dirnames). However, it seems the current walk implementation builds the entire file tree in advance before walking. Have you considered an iterative walk similar to os.walk to support sparse searches like this? I imagine this would require more network requests, but possibly save time and data requested if the search is sufficiently sparse. An iterative implementation might also be more consistent with users' expectation, coming from os.walk/Path.walk.

Thanks for this super useful library :)

clane9 avatar May 14 '25 13:05 clane9

We'd be open to making walk(top_down=True) be non-recursive so these kinds of filtering scenarios are more efficient.

pjbull avatar Aug 04 '25 20:08 pjbull