iis icon indicating copy to clipboard operation
iis copied to clipboard

Add optional compression support in avro2json workflow

Open marekhorst opened this issue 3 years ago • 0 comments

Currently avro2json workflow is responsible for translating avro input into json output.

It will be convenient, also for #918, to allow enabling compression by providing e.g. compression_method input parameter which could result in adding:

.option("compression", params.compressionMethod)

before .json() spark method producing json out of the Dataset.

Compression should be disabled when set to null, empty or $UNDEFINED$ value.

marekhorst avatar Aug 03 '22 10:08 marekhorst