guac
guac copied to clipboard
Create upper limit for batch size in batch ingestion
Batch ingestion results in performance improvements for moderately sized batches. Large batches have 2 possible causes for performance degradation:
- graphQL query document is too large, resulting in long times spent in assembling and parsing the request, assembling the response
- database backend needs to transfer all data to unwind, large batches could cause OOMs and retries (large transactions are also an issue)
We need to carefully profile and select batch size accordingly. Perhaps even provide a tuning knob so that deployments of GUAC can be optimized locally as needed
Still important but adding long-term since this tweakable later, and likely will be affected by future changes in API.