cloud-pipeline icon indicating copy to clipboard operation
cloud-pipeline copied to clipboard

Issue 733 Bulk request size limitation

Open Wedds opened this issue 6 years ago • 0 comments

This PR partially solves issue #733 . Request chunk splitting additionally covered by chunk memory size limitation besides elements count limitation. Elasticsearch can't index request with size over the value of its http.max_content_length property, so we are tracking bulk request size.

  • introduce elastic.request.limit.size.mb in application.properties;
  • monitor documents' size during chunk creation (execute bulk request if the next request added will cause limitation oversize; skip the files exceeding the limit).

Wedds avatar Oct 22 '19 15:10 Wedds