cloud-pipeline
cloud-pipeline copied to clipboard
Issue 733 Bulk request size limitation
This PR partially solves issue #733 .
Request chunk splitting additionally covered by chunk memory size limitation besides elements count limitation. Elasticsearch can't index request with size over the value of its http.max_content_length property, so we are tracking bulk request size.
- introduce
elastic.request.limit.size.mbin application.properties; - monitor documents' size during chunk creation (execute bulk request if the next request added will cause limitation oversize; skip the files exceeding the limit).