hudi
hudi copied to clipboard
apache
[HUDI-7228] Fix eager closure of log reader input streams with log record reader
Change Logs
Fix eager closure of log reader input streams with log record reader. While reading log blocks, we do make sure to close the actual content and release it for GC purposes. But the inputstream instantiated with log record reader is not closed. Fixing the same in this patch.
Impact
Fix eager closure of log reader input streams with log record reader. Memory usage when reading 100+ log files will be reduced to minimal.
Memory snapshot before this patch: 25 log files/25 log blocks were present in the file slice of interest.
Notice 25 instances of input streams open.
Memory snapshot after this patch.
Dump taken just after reading all log block headers, but before starting to deser any block for records.
Risk level (write none, low medium or high below)
If medium or high, explain what verification was done to mitigate the risks.
Documentation Update
Describe any necessary documentation update if there is any new feature, config, or user-facing change
- The config description must be updated if new configs are added or the default value of the configs are changed
- Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.
Contributor's checklist
- [ ] Read through contributor's guide
- [ ] Change Logs and Impact were stated clearly
- [ ] Adequate tests were added if applicable
- [ ] CI passed
I see many test failures the input stream of log reader can not be closed: https://github.com/apache/hudi/actions/runs/7228545731/job/19698162762?pr=10340#step:6:7650
@nsivabalan : Looks like the PR is already approved but needs fixing conflicts. Are you planning to merge this at later time ?
Yeah，it is already cherry-picked into 0.14.1 release, just resolve the conflicts and we are good to merge.
