iis
iis copied to clipboard
Add optional compression support in avro2json workflow
Currently avro2json workflow is responsible for translating avro input into json output.
It will be convenient, also for #918, to allow enabling compression by providing e.g. compression_method input parameter which could result in adding:
.option("compression", params.compressionMethod)
before .json() spark method producing json out of the Dataset.
Compression should be disabled when set to null, empty or $UNDEFINED$ value.