azkaban-plugins icon indicating copy to clipboard operation
azkaban-plugins copied to clipboard

Got error when executing Hive on Snappy supported Hadoop

Open changzong opened this issue 8 years ago • 1 comments

Hi, I just tried to run hive script with azkaban 3.0 with the hive plugin, and this error came up:

"Unable to get CompressorType for codec (org.apache.hadoop.io.compress.SnappyCodec). This is most likely due to missing native libraries for the codec."

My Hadoop supports SnappyCodec and every configuration in hadoop and azkaban plugin seem correct. I think the problem may come from missing native library configuration but I can't find any place to set. Any suggestion?

Thanks

changzong avatar Nov 25 '16 02:11 changzong

I just knew how this came up. The problem is from my tez configuration. I set tez.runtime.compress to true to compress my intermediate data and use SnappyCodec as the compression method. After I set the config to false, every thing succeeded. I still don't understand why this parameter cannot be used.

changzong avatar Nov 28 '16 07:11 changzong