hudi icon indicating copy to clipboard operation
hudi copied to clipboard

[WIP][HUDI-4449] Support DataSourceV2 Read for Spark3.2

Open southernriver opened this issue 3 years ago • 5 comments

Change Logs

NA

Impact

NA

Risk level (write none, low medium or high below)

medium

Contributor's checklist

  • [ ] Read through contributor's guide
  • [ ] Change Logs and Impact were stated clearly
  • [ ] Adequate tests were added if applicable
  • [ ] CI passed

southernriver avatar Aug 19 '22 01:08 southernriver

does this PR only aimed to support spark 3.2.0 not spark 3.2.1/3.2.2 or 3.3.0? or with following-up tasks to support other versions?

leesf avatar Aug 20 '22 10:08 leesf

@leesf thanks for your time to review this, I found that there are serveral problems with this pr, since upstream has revert v2 datasource, https://github.com/apache/hudi/blob/3adb5715310b372c2e0330102169c40d1c7448ba/hudi-spark-datasource/hudi-spark3.2.x/src/main/scala/org/apache/hudi/Spark3DefaultSource.scala#L28, I'll update soon

southernriver avatar Aug 22 '22 14:08 southernriver

@alexeykudinkin FYI

yihua avatar Aug 23 '22 18:08 yihua

@leesf @alexeykudinkin @XuQianJin-Stars Thanks for your time. I've been busy with other things these days, sorry for the late reply.

southernriver avatar Oct 31 '22 13:10 southernriver

CI report:

  • 8040b170bc2fd74dd9ada073b3b490baa8a771c0 Azure: FAILURE Azure: FAILURE
Bot commands @hudi-bot supports the following commands:
  • @hudi-bot run azure re-run the last Azure build

hudi-bot avatar Nov 05 '22 09:11 hudi-bot