data-prepper
data-prepper copied to clipboard
New OpenSearch API source implementation
Description
In order for DataPrepper to support all OpenSearch Document API(s), we need to build a new source similar to the existing http source. This pull request is intended to implement a new OpenSearch API source like opensearch_api
similar to http
source. This source should support the Document API Bulk.
This pull request includes the following:
- Path and HTTP methods:
- POST
_bulk
- POST
<index>/_bulk
- POST
- Optional URL parameters
-
pipeline
androuting
. (TODO:pipeline
parameter handling on Sink side)
-
- Multi-line JSON Bulk Request payload
Example pipeline configuration with opensearch_api
source looks like:
simple-sample-pipeline:
source:
opensearch_api:
path: "/opensearch"
port: 9202
sink:
...
Issues Resolved
Contributes to #248
Check List
- [X] New functionality includes testing.
- [ ] New functionality has a documentation issue. Please link to it in this PR.
- [X] New functionality has javadoc added
- [X] Commits are signed with a real name per the DCO
By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license. For more information on following Developer Certificate of Origin and signing off your commits, please check here.