Performance issue with directories full of many files

[patriziotomato]

Hi guys,

is anyone else encountering performance issues dealing with folders containing huge number of files? Even though I just want to list all directories (without any containing files), it seem to read also the files within the folders. This causes serious performance issue for us.

Thank you!

[davimatos]

Hi!

I don't know if it would be the same problem, but for about 10 days, I've been getting "Reduce your request rate. SlowDown" responses from Spaces, from Digital Ocean. Apparently I don't make excessive use of the S3 driver's resources, so this error was not supposed to happen. I don't know how to solve this problem.

[dmitriy-komarov]

Hi @patriziotomato @davimatos ,

Did you find a workaround for this?

[davimatos]

Hi @patriziotomato @davimatos ,

Did you find a workaround for this?

I did a deep analysis on the operations I was doing for my s3 drive, and I noticed that there were several "failed" operations, for example: I sent a delete operation to a file that no longer existed. I believe these bulk operations were causing this error.

I corrected these operations and stopped getting this message.

To improve, I also revised the way I stored the files, considering the good practices of prefixes, which helps in terms of performance and avoids reaching the limits of use. See at https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-prefixes.html