Unable to set Compression config when using AvroWriter

[richiesgr]

Hi I would like to add more on this because I check it now The problem is that Avro doesn't support Gzip compression by default. The problem is that Avro and the message writer both use the same configuration params: secor.compression.codec So if you put org.apache.hadoop.io.compress.GzipCodec you get an exception because Avro writer try to use it and fail If you put null (no compression) so the MessageWriter fail with an exception because it try to instance a class with that If put empty it fails because there you must put value

So I don't know what to put here because I try to Big query to read the files and don't thing it support something else than Gzip or uncompressed. and you can't set neither !!

Originally posted by @richiesgr in https://github.com/pinterest/secor/issues/482#issuecomment-735748818

[HenryCaiHaiying]

Richard,

If you need to set those two params separately, you can modify the secor code to introduce two separate params.

On Mon, Nov 30, 2020 at 4:14 AM Richard Grossman [email protected] wrote:

Hi I would like to add more on this because I check it now The problem is that Avro doesn't support Gzip compression by default. The problem is that Avro and the message writer both use the same configuration params: secor.compression.codec So if you put org.apache.hadoop.io.compress.GzipCodec you get an exception because Avro writer try to use it and fail If you put null (no compression) so the MessageWriter fail with an exception because it try to instance a class with that If put empty it fails because there you must put value

So I don't know what to put here because I try to Big query to read the files and don't thing it support something else than Gzip or uncompressed. and you can't set neither !!

Originally posted by @richiesgr https://github.com/richiesgr in #482 (comment) https://github.com/pinterest/secor/issues/482#issuecomment-735748818

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1725, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP74PTN7GEZNZSGCGKXTSSOECVANCNFSM4UHPPU4Q .

[richiesgr]

Hi Yes I've already made the modification. Don't know if it's something that can interest someone for a PR If yes I can push a PR

[HenryCaiHaiying]

I don't think it will hurt to introduce that parameter, feel free to submit a PR.

On Tue, Dec 1, 2020 at 1:01 AM Richard Grossman [email protected] wrote:

Hi Yes I've already made the modification. Don't know if it's something that can interest someone for a PR If yes I can push a PR

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/pinterest/secor/issues/1725#issuecomment-736326329, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABYJP74AZISWO3HE2JTZGR3SSSWINANCNFSM4UHPPU4Q .