hindsight icon indicating copy to clipboard operation
hindsight copied to clipboard

batch fetch/export process for `export` command

Open zeroXbrock opened this issue 2 years ago • 1 comments

hindsight export currently loads ALL entries from the source db into memory before dumping them into the destination.

This is reeeal slow on a large dataset (1+ months).

proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save

  • file export will need modification to append to the file, also thread-safe lock on the file ofc
  • db ops can likely be done w/ parallel connections, as there won't be any write conflicts coming from the source

zeroXbrock avatar Sep 07 '23 04:09 zeroXbrock

this should also fix the stalling effect @ end of export command

zeroXbrock avatar Sep 07 '23 04:09 zeroXbrock