data-prepper icon indicating copy to clipboard operation
data-prepper copied to clipboard

Support gzip encoded content in http source

Open graytaylor0 opened this issue 2 years ago • 1 comments

Is your feature request related to a problem? Please describe. The http source does not support compressed gzip data

Describe the solution you'd like

  • The http source should read gzip encoded data, decompress it, and translate it into individual Events. This logic will be repeated in the s3 source, and potentially other sources. It makes sense to have a plugin that can be reused by different sources to perform this logic.

Additional context

  • Another request for the otel-trace-source to support compressed data was made in this issue (https://github.com/opensearch-project/data-prepper/issues/1152)
  • The s3 source design outlines support for gzip encoded data (https://github.com/opensearch-project/data-prepper/issues/251)

graytaylor0 avatar May 16 '22 15:05 graytaylor0

This issue should share the same solution as in #1176 .

chenqi0805 avatar May 17 '22 20:05 chenqi0805