TileDB icon indicating copy to clipboard operation
TileDB copied to clipboard

Better rate-limiting for S3 requests on SlowDown exceptions

Open tdenniston opened this issue 6 years ago • 0 comments

We've observed SlowDown S3 errors when issuing many simultaneous read queries on large arrays, e.g.:

what():  [TileDB::S3] Error: Cannot retrieve S3 object size; Error while listing file s3://tiledb-arrays/array-a/__56761709d992432098cb9b18c516fc8f_1553215612527/__fragment_metadata.tdb"
Exception:  SlowDown
Error message:  Please reduce your request rate.

Need to investigate better ways to handle this:

  • Custom backoff logic
  • Better-tuned SDK config params for retries
  • ?

tdenniston avatar Mar 22 '19 18:03 tdenniston