guac icon indicating copy to clipboard operation
guac copied to clipboard

Create upper limit for batch size in batch ingestion

Open mihaimaruseac opened this issue 2 years ago • 1 comments

Batch ingestion results in performance improvements for moderately sized batches. Large batches have 2 possible causes for performance degradation:

  • graphQL query document is too large, resulting in long times spent in assembling and parsing the request, assembling the response
  • database backend needs to transfer all data to unwind, large batches could cause OOMs and retries (large transactions are also an issue)

We need to carefully profile and select batch size accordingly. Perhaps even provide a tuning knob so that deployments of GUAC can be optimized locally as needed

mihaimaruseac avatar Jul 01 '23 12:07 mihaimaruseac

Still important but adding long-term since this tweakable later, and likely will be affected by future changes in API.

lumjjb avatar Apr 02 '24 23:04 lumjjb