[HUDI-7244] Ensure HoodieFileGroupReader.close() is called in spark

[jonvex]

Change Logs

iter.map as well as iterator.toScala were not calling close. We now don't use those.

Impact

Don't leak memory (at least here)

Risk level (write none, low medium or high below)

low

Documentation Update

N/A

Contributor's checklist

• [ ] Read through contributor's guide
• [ ] Change Logs and Impact were stated clearly
• [ ] Adequate tests were added if applicable
• [ ] CI passed

[linliu-code]

@jonvex , the tests are failing.

[danny0405]

Looks promising, the OOM is gone, the test failures look not related.

[linliu-code]

Did we chase down every reader (base files, log files, iterator) part or the new file group and ensured this is the only gap ? @jonvex @linliu-code

I haven't. I will try to do that later.

[jonvex]

Did we chase down every reader (base files, log files, iterator) part or the new file group and ensured this is the only gap ? @jonvex @linliu-code

I haven't. I will try to do that later.

I checked all the log blocks and the reader context. Could you check the record buffers?

[hudi-bot]

CI report:

• 33a87e77b985a8fd3fe0a6a997059ee20fbedb8b UNKNOWN
• 9819ca4db7b4ab9f2476aecc753e3fcc09c7cb7a Azure: FAILURE

Bot commands

@hudi-bot supports the following commands:

• @hudi-bot run azure re-run the last Azure build

[jonvex]

Azure CI is passsing: https://dev.azure.com/apache-hudi-ci-org/785b6ef4-2f42-4a89-8f0e-5f0d7039a0cc/_build/results?buildId=21791