DBSubsetter
DBSubsetter copied to clipboard
Consider query size cap or batch fetching for safety
Currently there is no maximum size on the amount of data we could pull into memory at once. It's dependent on:
- What the user specifies as base query
- What the cardinality of children to parents is (for instance, if one parent row has 5 billion children rows, we might pull all 5 billion into memory at once)
There may be some way to give more safety around memory use. For instance, by putting a cap on rows fetched, by using jdbc batches for selects, etc.