hudi icon indicating copy to clipboard operation
hudi copied to clipboard

Published 20 hours ago • verified publisher icon apache

[HUDI-7718] Use source profile in HoodieIncrSource

Open vinishjail97 opened this issue 9 months ago • 1 comments

Change Logs

Similar to KafkaSource, the source profile populated in StreamContext can be used for better parallelism and instead of using a static value for numInstantsPerFetch, a dynamic value generated based on heuristics can be used. This PR needs to be merged after [https://github.com/apache/hudi/pull/10918]

Impact

A new constructor added for HoodieIncrSource for emitting metrics related to source parallelism and source bytes ingested.

Risk level (write none, low medium or high below)

Medium

Documentation Update

None.

Contributor's checklist

  • [x] Read through contributor's guide
  • [x] Change Logs and Impact were stated clearly
  • [x] Adequate tests were added if applicable
  • [x] CI passed

vinishjail97 avatar May 06 '24 09:05 vinishjail97

CI report:

  • 4976e93e50e459d2fb30734a2f964135f6eef4a8 Azure: SUCCESS
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar May 06 '24 12:05 hudi-bot

This one can be closed, duplicate of this PR. https://github.com/apache/hudi/pull/11175

vinishjail97 avatar Sep 13 '24 12:09 vinishjail97