parquet-format

parquet-format copied to clipboard

Apart from the original authors' great work, Apache projects should not mention any contributors in particular in the POM. ### Jira - [ ] My PR addresses the following [Parquet...

Fokko

Register a MIME type for the Parquet format.

15

There is currently  no MIME type registered for Parquet.  Perhaps this is intentional. If it is not intentional, I suggest steps be taken to register a MIME type with IANA.    ...

asfimport

Parquet spec (parquet.thrift) is inconsistent w.r.t. ColumnIndex + NaNs

10

Currently, the specification of `ColumnIndex` in `parquet.thrift` is inconsistent, leading to cases where it is impossible to create a parquet file that is conforming to the spec. The problem is...

asfimport

[Format] Encoding spec incorrect for dictionary fallback

3

The spec for DICTIONARY_ENCODING states that: > If the dictionary grows too big, whether in size or number of distinct values, the encoding will fall back to the plain encoding....

asfimport

The Parquet Spec doesn't specify whether multiple columns are allowed to have the same name.

2

The parquet format specification doesn't say whether a Parquet file having columns with the same name (in the same group node, so really exactly the same name) is valid. I.e.,...

asfimport

parquet-format and parquet-format-structures defines Util with inconsitent methods provided

2

I have been running into a bug due to `parquet-format` and `parquet-format-structures` both defining the `org.apache.parquet.format.Util` class but doing so inconsistently. Examples of this are several methods which include a...

asfimport

BloomFilter and float point is ambiguous

5

Currently, our Parquet can use BloomFilter for any physical types. However, when BloomFilter apply on float: 1. What does +0 -0 means? Are they equal? 1. Should qNaN sNaN written...

asfimport

Parquet Filter predicate storing nested string causing OOM's

3

Each Instance of ColumnFilterPredicate stores the filter values in toString variable eagerly. Which is not useful ```java static abstract class ColumnFilterPredicate implements FilterPredicate, Serializable { private final Column column; private...

asfimport

Support gorilla encoding for float numbers

1

[Gorilla](https://www.vldb.org/pvldb/vol8/p1816-teller.pdf) is a de facto encoding algorithm for float numbers, it has been used by many time series database such as InfluxDB, TimescaleDB for a while. For now Parquet only...

asfimport

I often needs to create tens of milliions of small dataframes and save them into parquet files. all these dataframes have the same column and index information. and normally they...

asfimport