[HUDI-7228] Fix eager closure of log reader input streams with log record reader (0.x branch)

[nsivabalan]

Change Logs

Fix eager closure of log reader input streams with log record reader. While reading log blocks, we do make sure to close the actual content and release it for GC purposes. But the inputstream instantiated with log record reader is not closed. Fixing the same in this patch.

Impact

Fix eager closure of log reader input streams with log record reader. Memory usage when reading 100+ log files will be reduced to minimal.

Memory snapshot before this patch: 25 log files/25 log blocks were present in the file slice of interest.

Notice 25 instances of input streams open.

Memory snapshot after this patch. Dump taken just after reading all log block headers, but before starting to deser any block for records.

Risk level (write none, low medium or high below)

If medium or high, explain what verification was done to mitigate the risks.

Documentation Update

Describe any necessary documentation update if there is any new feature, config, or user-facing change

• The config description must be updated if new configs are added or the default value of the configs are changed
• Any new feature or user-facing change requires updating the Hudi website. Please create a Jira ticket, attach the ticket number here and follow the instruction to make changes to the website.

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[linliu-code]

I think this also failed the master. Why just the 0.X branch? Meanwhile, when is this bug introduced? I start to notice the OOM problem recently.

[nsivabalan]

@linliu-code : yes, we do have a fix for master here here https://github.com/apache/hudi/pull/10340

[hudi-bot]

CI report:

• 1a2b5d4c84b5fae678a8e0f0e4edcac870cb6e8f Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build