FSSpecFileLister slow warmup with many files

[platers]

🚀 The feature

It would be great if FSSpecFileLister could iteratively list files instead of preloading them all.

Motivation, pitch

With a cloud provider, listing all the files in a directory is slow and expensive. In my GCS dir with 200k files FSSpecFileLister takes forever to return a single url.

It would be great if FSSpecFileLister could iteratively list files instead of preloading them all.

Alternatives

No response

Additional context

fsspec seems to support returning a generator via walk().

https://filesystem-spec.readthedocs.io/en/latest/api.html#fsspec.spec.AbstractFileSystem.walk