google-cloud-go
google-cloud-go copied to clipboard
feat(storage/dataflux): run worksteal listing parallel to sequential listing
feat: run worksteal listing parallel to sequential listing Dataflux fast-listing leverages worksteal algorithm to quickly list objects in a bucket by running several parallel processes. Worksteal algorithm splits a given namespace into multiple ranges for multiple workers(goroutines) to list objects in gcs bucket in parallel.
Adding logic to list next page in worksteal listing and basic unit tests that uses storage/emulator.
Fixes https://github.com/googleapis/google-cloud-go/issues/10731