adlfs icon indicating copy to clipboard operation
adlfs copied to clipboard

Published 20 hours ago • verified publisher icon fsspec

Limit concurrency

Open hayesgb opened this issue 2 years ago • 2 comments

Picking up of #288

hayesgb avatar Jul 23 '22 00:07 hayesgb

Thanks for picking this up!

One thing I've learned about since starting this PR, azure-storage-blob already uses the max_concurrency keyword for something that's slightly different. The BlobClient.upload_blob and BlobClient.download_blob use max_concurrency to control the number of threads / concurrent tasks to run. It's similar, but slightly difference since it controls concurrency / parallelism on a single blob operation. This PR was targeting operations on many blobs.

We might want to consider a different keyword (max_clients? max_blob_clients?) so that we can pass through max_concurrency for parallelism on single-blob operations.

TomAugspurger avatar Jul 23 '22 01:07 TomAugspurger

Hey @TomAugspurger -- That sounds like a great idea.

Unit tests were passing in my local.

hayesgb avatar Jul 25 '22 03:07 hayesgb