hindsight
hindsight copied to clipboard
batch fetch/export process for `export` command
hindsight export currently loads ALL entries from the source db into memory before dumping them into the destination.
This is reeeal slow on a large dataset (1+ months).
proposed solution: query 1000 arbs at a time, export them as you go; maximize parallelization of load/save
- file export will need modification to append to the file, also thread-safe lock on the file ofc
- db ops can likely be done w/ parallel connections, as there won't be any write conflicts coming from the source
this should also fix the stalling effect @ end of export command