hudi [HUDI-7501] Use source profile for S3 and GCS sources

[HUDI-7501] Use source profile for S3 and GCS sources

Open vinishjail97 opened this issue 11 months ago • 3 comments

Change Logs

Use SourceProfile in S3/GCS sources for the sourceLimit and numPartitions. Instead of relying on the sourceLimit passed in from static config use the one provided by the source profile.

Impact

No impact to existing public constructors, the change is backwards compatible.

Risk level (write none, low medium or high below)

Medium

Documentation Update

None, this is just adding a new public constructor to include StreamContext as a parameter in S3/GCS sources.

Contributor's checklist

[x] Read through contributor's guide
[x] Change Logs and Impact were stated clearly
[x] Adequate tests were added if applicable
[ ] CI passed

Mar 13 '24 19:03 vinishjail97

Is it possible to get a test coverage for the source classes touched and share the report.

Generally, GcsEventsHoodieIncrSource and S3EventsHoodieIncrSource does not have any function tests right. so, we rely heavily on mocked UTs.

GcsEventsHoodieIncrSource, S3EventsHoodieIncrSource, CloudDataFetcher and CloudObjectsSelectorCommon have decent coverage