iceberg icon indicating copy to clipboard operation
iceberg copied to clipboard

Build: Fix BrotliCodec class not found failure when using brotli as compression codec

Open singhpk234 opened this issue 2 years ago • 1 comments

About the change

presently when using brotli as compression codec for parquet it fails with

Caused by: org.apache.parquet.hadoop.BadConfigurationException: Class org.apache.hadoop.io.compress.BrotliCodec was not found
	at org.apache.parquet.hadoop.CodecFactory.getCodec(CodecFactory.java:243)
	at org.apache.parquet.hadoop.CodecFactory$HeapBytesCompressor.<init>(CodecFactory.java:144)
	at org.apache.parquet.hadoop.CodecFactory.createCompressor(CodecFactory.java:208)
	at org.apache.parquet.hadoop.CodecFactory.getCompressor(CodecFactory.java:191)

this just makes sure BrotliCodec is available in class path so that it can be loaded and doesn't fails with the error above.

Alternatively we could also mention in iceberg to install BrotliCodec when selecting brotli as the compression, somewhat spark does (https://spark.apache.org/docs/latest/sql-data-sources-parquet.html, ref: brotli requires BrotliCodec to be installed) rather than taking a new dependency .

cc @rdblue @jackye1995

singhpk234 avatar May 04 '23 20:05 singhpk234

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the [email protected] list. Thank you for your contributions.

github-actions[bot] avatar Aug 29 '24 00:08 github-actions[bot]

This pull request has been closed due to lack of activity. This is not a judgement on the merit of the PR in any way. It is just a way of keeping the PR queue manageable. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.

github-actions[bot] avatar Sep 05 '24 00:09 github-actions[bot]