opensearch-project
[DOC] Clarification regarding data prepper sinks.
What do you want to do?
- [X] Request a change to existing documentation
- [X] Add new documentation
- [ ] Report a technical problem with the documentation
- [ ] Other
Tell us about your request. Provide a summary of the request.
Someone was asking about whether data prepper can "handle" apache avro data, and found that the documentation wasn't entirely clear. avro is listed as a codec for data prepper, but refers to it "most efficiently being used" in an S3 sink. Could we add a paragraph or so about how it can be used outside of an S3 sink?
Also - it seems to have some weird formatting oddities that make it a little hard to skim. See screenshots.
*Version: List the OpenSearch version to which this issue applies, e.g. 2.14, 2.12--2.14, or all.
2.15
What other resources are available? Provide links to related issues, POCs, steps for testing, etc.
@dlvenable - Can you please comment on this? Here is the link: https://opensearch.org/docs/latest/data-prepper/common-use-cases/codec-processor-combinations/#avro
Regarding the original question, Data Prepper can read Avro from S3 and write Avro to S3.
Regarding the documentation, we should revisit this page. The original intention was to clarify when a user should use a codec versus a processor for parsing input data.
I might reword this as:
Apache Avro is an open-source serialization format for record data. When reading Avro data you should use the
avrocodec.
I also noticed some question comments about Parquet.
Apache Parquet is a columnar storage format built for Hadoop. It is most efficient without the use of a codec. Positive results, however, can be achieved when it’s configured with S3 Select.
Perhaps this should say:
Apache Parquet is a columnar storage format built for Hadoop. Pipeline authors can use the
parquetcodec to read Parquet data directly from the S3 object. This will retrieve all data from Parquet. An alternative is to use S3 Select instead of the codec. In this case, S3 Select will parse the Parquet file directly (additional S3 charges apply). This can be more efficient if you are filtering or loading a subset of data.