spark icon indicating copy to clipboard operation
spark copied to clipboard

Published 20 hours ago • verified publisher icon apache

[SPARK-39799][SQL] DataSourceV2: View catalog interface

Open jzhuge opened this issue 2 years ago • 4 comments

What changes were proposed in this pull request?

ViewCatalog API described in SPIP.

Why are the changes needed?

First step towards DataSourceV2 view support.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

N/A

jzhuge avatar Aug 17 '22 16:08 jzhuge

cc @dongjoon-hyun who had comments on the SPIP

holdenk avatar Aug 18 '22 18:08 holdenk

Thank you for pinging me, @holdenk .

cc @viirya , @sunchao , @huaxingao , @aokolnychyi , @RussellSpitzer

dongjoon-hyun avatar Aug 18 '22 18:08 dongjoon-hyun

Find this message strange from https://github.com/jzhuge/spark/runs/7886846067?check_suite_focus=true:

[info] Found the following changed modules: catalyst, hive-thriftserver

This PR does not change module hive-thriftserver.

jzhuge avatar Aug 18 '22 19:08 jzhuge

Interesting, the failure in CliSuite seems like a red-herring (e.g. that text is present), maybe try a re-base + re-run?

holdenk avatar Aug 22 '22 22:08 holdenk

pending CI and any other concerns from others I plan to merge this on Friday.

holdenk avatar Nov 03 '22 02:11 holdenk

Looking at error:

SparkThrowableSuite.Error classes are correctly formatted

jzhuge avatar Nov 03 '22 03:11 jzhuge

Puzzled by this pyspark test failures. Seems unrelated.

[info] compiling 25 Scala sources to /__w/spark/spark/connector/docker-integration-tests/target/scala-2.12/test-classes ...
[error] /__w/spark/spark/sql/hive/src/test/scala/org/apache/spark/sql/HiveCharVarcharTestSuite.scala:49:13: exception during macro expansion: 
[error] java.util.MissingResourceException: Can't find bundle for base name org.scalactic.ScalacticBundle, locale en_US

jzhuge avatar Nov 05 '22 19:11 jzhuge

Current diff LGTM. We've been testing this out internally (including changes from the umbrella PR as well) and with the recent changes I think this PR looks like it should set us up for success with the rest of the feature.

xkrogen avatar Nov 07 '22 18:11 xkrogen

Looking at error:

SparkThrowableSuite.Error classes are correctly formatted

Fixed

jzhuge avatar Nov 07 '22 18:11 jzhuge

Hi @jzhuge @holdenk, can we merge this PR? Or is there any other concerns?

ljfgem avatar Nov 10 '22 20:11 ljfgem

LGTM I'll merge this now to the current dev branch.

holdenk avatar Nov 16 '22 16:11 holdenk

Thanks @holdenk, @wmoustafa, @xkrogen, @ljfgem for the reviews!

jzhuge avatar Nov 16 '22 17:11 jzhuge