dfuse-eosio icon indicating copy to clipboard operation
dfuse-eosio copied to clipboard

Feature: optimize deletes in merger when using s3 buckets for one-block storage

Open matthewdarwin opened this issue 4 years ago • 0 comments

If you are running mindreader with the mindreader-oneblock-suffix option, and also running multiple mindreaders at once (for redundancy), then you end up with a lot of one-block files.

Currently the merger deletes these files one at time, which is not very efficient.

Bundle deletes together or use a pattern based delete mechanism to have less load on the s3 storage backend.

matthewdarwin avatar Sep 09 '20 06:09 matthewdarwin