elephant-bird
elephant-bird copied to clipboard
twitter
Twitter's collection of LZO and Protocol Buffer-related Hadoop, Pig, Hive, and HBase code.
Proposed resolution to #426, which describes how FileInputFormat's listStatus is slow on S3 for input paths spanning many files.
When iterating over input splits via DeprecatedInputFormatWrapper, DeprecatedInputFormatWrapper.java always calls mifcReader.setKeyValue(key, value) before nextValue is invoked which can call through to setKeyValue in CompositeRecordReader.java. setKeyValue requires that the currentRecordReader instance...
so that RCFilePigStorage works with hive 0.14 (and is backwards compatible with previous versions) Because 'public int getCount()' was removed from org.apache.hadoop.hive.serde2.ByteStream.Output in hive 0.14, and getLength() is available from...
@isnotinvain Attempt to make numRecordsPerBlock configurable. When testing locally there's a strange failure for pig even when building from master (at line 166 of TestThriftToPig.java): ``` Failed tests: test(com.twitter.elephantbird.pig.util.TestThriftToPig): expected:...
In some scenario, LzoThriftBlock file may be corrupt. For example, in our company, we consume kafka data and sink to HDFS as LzoThriftBlock format using a flink streaming job. If...
Hi, now the developing `version` of master is `4.17-SNAPSHOT` and the latest release version is `4.17`, so the latest snapshot is behind the released version. From the commits, we can...
In the Wiki page, the link 'Elephant Bird Lucene: Querying Indexes' has broken.
create external table tmp_test1 row format serde 'org.apache.hadoop.hive.serde2.thrift.ThriftDeserializer' with serdeproperties ( "serialization.class"="xxxx", "serialization.format"="org.apache.thrift.protocol.TCompactProtocol") stored as parquet; I create hive table successfully, but load data got problem: "Failed with exception java.io.IOException:java.lang.ClassCastException:...
I am reading protobuf data using ProtobufPigLoader, it will fail when my proto have nested defination like following example, the error log is attched. I didn't set any value on...
Hi, Im desperately trying to install the git version of elephant-bird; latest thrift+protobuf were installed accordingly. I executed mvn package and got error below [ERROR] Failed to execute goal com.github.igor-petruk.protobuf:protobuf-maven-plugin:0.6.5:run...
