hadoop icon indicating copy to clipboard operation
hadoop copied to clipboard

HADOOP-19140. [ABFS, S3A] Add IORateLimiter API

Open steveloughran opened this issue 10 months ago • 3 comments

Adds an API (pulled from #6596) to allow callers to request IO capacity for an named operation with optional source and dest paths.

The first use of this would be the bulk delete operation of #6494; there'd be some throttling within the s3a code which set max # of writes per bucket and for the bulk delete the caller would ask for as many as there were entries.

Added new store operations for delete_bulk and delete_dir

How was this patch tested?

New tests.

For code changes:

  • [X] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
  • [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
  • [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
  • [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?

steveloughran avatar Apr 03 '24 16:04 steveloughran