data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

New OpenSearch API source implementation

Open sb2k16 opened this issue 8 months ago • 0 comments

Description

In order for DataPrepper to support all OpenSearch Document API(s), we need to build a new source similar to the existing http source. This pull request is intended to implement a new OpenSearch API source like opensearch_api similar to http source. This source should support the Document API Bulk.

This pull request includes the following:

  1. Path and HTTP methods:
    • POST _bulk
    • POST <index>/_bulk
  2. Optional URL parameters
    • pipeline and routing. (TODO: pipeline parameter handling on Sink side)
  3. Multi-line JSON Bulk Request payload

Example pipeline configuration with opensearch_api source looks like:

simple-sample-pipeline:
  source:
    opensearch_api:
      path: "/opensearch"
      port: 9202
  sink:
   ...

Issues Resolved

Contributes to #248

Check List

  • [X] New functionality includes testing.
  • [ ] New functionality has a documentation issue. Please link to it in this PR.
  • [X] New functionality has javadoc added
  • [X] Commits are signed with a real name per the DCO

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.

sb2k16 avatar Jun 05 '24 16:06 sb2k16