data-prepper
data-prepper copied to clipboard
Support gzip encoded content in http source
Is your feature request related to a problem? Please describe.
The http
source does not support compressed gzip data
Describe the solution you'd like
- The
http
source should read gzip encoded data, decompress it, and translate it into individualEvents
. This logic will be repeated in thes3
source, and potentially other sources. It makes sense to have a plugin that can be reused by different sources to perform this logic.
Additional context
- Another request for the
otel-trace-source
to support compressed data was made in this issue (https://github.com/opensearch-project/data-prepper/issues/1152) - The
s3
source design outlines support forgzip
encoded data (https://github.com/opensearch-project/data-prepper/issues/251)
This issue should share the same solution as in #1176 .