orc icon indicating copy to clipboard operation
orc copied to clipboard

ORC-1286: [C++] replace DataBuffer with BlockBuffer in class BufferedOutputStream

Open coderex2522 opened this issue 2 years ago • 2 comments

What changes were proposed in this pull request?

This PR can solve the huge memory taken by BufferedOutputStream and refactor the write data logic in class CompressionBase.

Why are the changes needed?

This patch use BlockBuffer to replace DataBuffer of class BufferedOutputStream in order to solve the issue.

How was this patch tested?

The UTs in TestBufferedOutputStream.cc and TestCompression.cc can cover this patch.

coderex2522 avatar Oct 12 '22 04:10 coderex2522

@coderex2522 . Could you use a new JIRA instead of using PART 1 and PART II? Apache community uses ORC JIRA ID for trace-ability.

@dongjoon-hyun Thank you for your suggestion. I create a new JIRA ORC-1286.

coderex2522 avatar Oct 12 '22 09:10 coderex2522

Thank you for updating the PR and creating JIRA, @coderex2522 .

dongjoon-hyun avatar Oct 13 '22 15:10 dongjoon-hyun

Merged to main for Apache ORC 1.9.

cc @williamhyun , too

dongjoon-hyun avatar Oct 26 '22 16:10 dongjoon-hyun