hudi
hudi copied to clipboard
[HUDI-7718] Use source profile in HoodieIncrSource
Change Logs
Similar to KafkaSource
, the source profile populated in StreamContext
can be used for better parallelism and instead of using a static value for numInstantsPerFetch, a dynamic value generated based on heuristics can be used. This PR needs to be merged after [https://github.com/apache/hudi/pull/10918]
Impact
A new constructor added for HoodieIncrSource for emitting metrics related to source parallelism and source bytes ingested.
Risk level (write none, low medium or high below)
Medium
Documentation Update
None.
Contributor's checklist
- [x] Read through contributor's guide
- [x] Change Logs and Impact were stated clearly
- [x] Adequate tests were added if applicable
- [x] CI passed