presto icon indicating copy to clipboard operation
presto copied to clipboard

Reduce memory usage of StreamDescriptor in Flat Maps

Open arunthirupathi opened this issue 2 years ago • 0 comments

Flat Maps StreamDescriptor consume large amount of memory.

  1. Make additional sequences point to the base sequence to avoid duplicating references.
  2. Do not store nested streams, as they are computed, stored and retrieved only once in the constructor of the reader.

Should reduce typical flat map reader memory consumption by about 10-15 MB.

Test plan - Will add new tests in a modified commit.

== NO RELEASE NOTE ==

arunthirupathi avatar Aug 10 '22 15:08 arunthirupathi