google-cloud-go icon indicating copy to clipboard operation
google-cloud-go copied to clipboard

feat(storage/dataflux): run worksteal listing parallel to sequential listing

Open akansha1812 opened this issue 1 year ago • 0 comments

feat: run worksteal listing parallel to sequential listing Dataflux fast-listing leverages worksteal algorithm to quickly list objects in a bucket by running several parallel processes. Worksteal algorithm splits a given namespace into multiple ranges for multiple workers(goroutines) to list objects in gcs bucket in parallel.

Adding logic to list next page in worksteal listing and basic unit tests that uses storage/emulator.

Fixes https://github.com/googleapis/google-cloud-go/issues/10731

akansha1812 avatar Oct 08 '24 19:10 akansha1812