DBSubsetter icon indicating copy to clipboard operation
DBSubsetter copied to clipboard

Consider query size cap or batch fetching for safety

Open bluerogue251 opened this issue 5 years ago • 0 comments

Currently there is no maximum size on the amount of data we could pull into memory at once. It's dependent on:

  • What the user specifies as base query
  • What the cardinality of children to parents is (for instance, if one parent row has 5 billion children rows, we might pull all 5 billion into memory at once)

There may be some way to give more safety around memory use. For instance, by putting a cap on rows fetched, by using jdbc batches for selects, etc.

bluerogue251 avatar Nov 03 '19 15:11 bluerogue251