Iterative walk for sparsely traversing large nested directories
I'm using CloudPath.walk to search a large nested directory, but I'm only descending into a very small subset of subdirectories (by removing elements from dirnames). However, it seems the current walk implementation builds the entire file tree in advance before walking. Have you considered an iterative walk similar to os.walk to support sparse searches like this? I imagine this would require more network requests, but possibly save time and data requested if the search is sufficiently sparse. An iterative implementation might also be more consistent with users' expectation, coming from os.walk/Path.walk.
Thanks for this super useful library :)
We'd be open to making walk(top_down=True) be non-recursive so these kinds of filtering scenarios are more efficient.