parquet-java icon indicating copy to clipboard operation
parquet-java copied to clipboard

Reduce memory footprint for nulls caching

Open asfimport opened this issue 8 years ago • 0 comments

Scenario: There is a schema with many optional groups, e.g.

message example {
  required binary id (UTF8);
  optional group a1 (LIST) {
    repeated int64 array;
  }
  optional group a2 (LIST) {
    repeated int64 array;
  }
...
  optional group aN (LIST) {
    repeated int64 array;
  }
}

Many records without optional parameters are written. In this case groupNullCache will contain many elements which all are zeros.

Reporter: Boris Molodenkov

Related issues:

Note: This issue was originally created as PARQUET-1119. Please see the migration documentation for further details.

asfimport avatar Oct 01 '17 20:10 asfimport