hudi icon indicating copy to clipboard operation
hudi copied to clipboard

Published 20 hours ago • verified publisher icon apache

[HUDI-4881] Push down filters if possible when syncing partitions to Hive

Open boneanxs opened this issue 3 years ago • 1 comments
trafficstars

Change Logs

If possible, try to push down filters to Hive to avoid fetching too many partitions.

Impact

Describe any public API or user-facing feature change or any performance impact.

Risk level: low

Choose one. If medium or high, explain what verification was done to mitigate the risks.

Contributor's checklist

  • [ ] Read through contributor's guide
  • [ ] Change Logs and Impact were stated clearly
  • [ ] Adequate tests were added if applicable
  • [ ] CI passed

boneanxs avatar Sep 20 '22 11:09 boneanxs

Canceling the CI run to prioritize release blockers. Apologies.

codope avatar Sep 28 '22 05:09 codope

@hudi-bot run azure

boneanxs avatar Sep 29 '22 08:09 boneanxs

@codope @yihua @alexeykudinkin @xushiyan Hi, could you plz take a look this improvement?

boneanxs avatar Sep 30 '22 01:09 boneanxs

Gentle ping @codope @yihua @xushiyan

boneanxs avatar Oct 17 '22 06:10 boneanxs

@boneanxs apologies for the delay. will check soon

xushiyan avatar Oct 17 '22 06:10 xushiyan

@alexeykudinkin @xushiyan could you please review the new commit? The test failure is 137, not relate to this pr.

boneanxs avatar Nov 04 '22 01:11 boneanxs

@hudi-bot run azure

boneanxs avatar Nov 04 '22 01:11 boneanxs

CI report:

  • 81f856d99da09e5a9438fad2a0d111bc9062aba4 Azure: FAILURE Azure: SUCCESS
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar Nov 04 '22 04:11 hudi-bot

@hudi-bot run azure

boneanxs avatar Nov 22 '22 13:11 boneanxs

CI report:

  • 18279aff65227373e172951404c192de6846cf59 Azure: SUCCESS
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar Nov 23 '22 09:11 hudi-bot

@alexeykudinkin Hey all comments are addressed, please take a look.

boneanxs avatar Nov 24 '22 00:11 boneanxs

@boneanxs i approved it. @xushiyan any more comments from your end?

alexeykudinkin avatar Nov 28 '22 20:11 alexeykudinkin

@xushiyan gentle ping...

boneanxs avatar Dec 09 '22 06:12 boneanxs

@boneanxs sorry for the delay. A follow-up suggestion: the partition filter and expression classes look very generic and we can consider putting them in sync-common for other catalog sync implementations, which support partition sync too

xushiyan avatar Dec 09 '22 19:12 xushiyan

@xushiyan Thanks for merging this. Sure, I will do this in a follow-up pr.

boneanxs avatar Dec 10 '22 01:12 boneanxs