Gang Wu comments

Results 304 comments of


                                            Gang Wu

PARQUET-2196: Support LZ4_RAW codec

I have changed the approach of interop test to follow encryption which downloads test files from parquet-testing repo and verifies the data decompressed and decoded from them. In addition, the...

Huge memory taken for each field when exporting

We may replace the DataBuffer by a new Buffer implementation with a much smarter memory management to automatically grow and shrink its size according to actual usage. This management can...

Huge memory taken for each field when exporting

I have created a JIRA to track the progress: https://issues.apache.org/jira/browse/ORC-1280

allow read old parquet file which is maked by old api with old avro version which allow wrong default value in schema

Thanks @wwang-talend for opening the PR! Could you create a JIRA issue for this?

Java impl does not fully adhere to the usage contract of `ColumnVector.noNulls` and `ColumnVector.isNull`

IMHO, it is safe to make the change if the writer and reader can promise the contract to the downstream consumers (e.g. Apache Spark and Apache Hive). The ORC java...

DO NOT MERGE: Expand BYTE_STREAM_SPLIT to support FIXED_LEN_BYTE_ARRAY, INT32 and INT64

@pitrou Do you want to have this in the 1.14.0 release?

ORC-1700: Write parquet decimal type data in Benchmark using `FIXED_LEN_BYTE_ARRAY` type

Should we use INT32 and INT64 for decimals where applicable?

ORC-1669: [C++] Upgrade libhdfs

``` CMake Error at /usr/share/cmake-3.18/Modules/FindPackageHandleStandardArgs.cmake:165 (message): Could NOT find Boost (missing: Boost_INCLUDE_DIR date_time) (Required is at least version "1.72.0") ``` It seems that the latest version of libhdfspp has a...

ORC-1669: [C++] Upgrade libhdfs

That sounds good. Let me mark relevant code as deprecated first.

ORC-1662: [C++] Upgrade protobuf to 3.21.12

Thanks for the PR! Unfortunately the new protobuf version is not compatible to compile libhdfspp. Let me investigate this.