spark icon indicating copy to clipboard operation
spark copied to clipboard

Published 20 hours ago • verified publisher icon apache

[SPARK-31357][SQL][WIP] Catalog API for view metadata

Open jzhuge opened this issue 3 years ago • 10 comments
trafficstars

What changes were proposed in this pull request?

This PR adds support to load, create, alter, and drop views in DataSource V2 catalogs.

This is an umbrella PR that combines the following commits that probably should be merged one by one.

  • View catalog interface
  • View substitution rule
  • Create view DDL
  • View SQL DDLs
  • Caching for ViewCatalog

Why are the changes needed?

Support views stored in DataSourceV2 catalogs. Details in SPIP.

Does this PR introduce any user-facing change?

  • Support views from DataSource V2 catalogs in SQL
  • Support views from DataSource V2 catalogs in DataFrame API

How was this patch tested?

New unit tests

Regression

  • DDLParserSuite
  • PlanResolutionSuite
  • DataSourceV2SQLSuite

jzhuge avatar Feb 23 '22 21:02 jzhuge

@jzhuge The PR as-is seems to only contain the view catalog interface, not the other 3 facets that you mentioned. Is this intentional?

xkrogen avatar Mar 01 '22 23:03 xkrogen

Sorry, not added yet.

On Tue, Mar 1, 2022 at 3:07 PM Erik Krogen @.***> wrote:

@jzhuge https://github.com/jzhuge The PR as-is seems to only contain the view catalog interface, not the other 3 facets that you mentioned. Is this intentional?

— Reply to this email directly, view it on GitHub https://github.com/apache/spark/pull/35636#issuecomment-1055950260, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOL5JBTAQ7T2PPHR73HSFLU52PJLANCNFSM5PFSWFAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- John Zhuge

jzhuge avatar Mar 01 '22 23:03 jzhuge

https://github.com/apache/spark/pull/28147 PR available? Why submit a new PR?

melin avatar Mar 11 '22 09:03 melin

@melin #28147 was marked stale and closed.

jzhuge avatar Mar 24 '22 16:03 jzhuge

Retest please

jzhuge avatar Mar 31 '22 17:03 jzhuge

Does this PR need to update ResolveSessionCatalog? Currently, CreateView only supports V1 catalog.

frankliee avatar May 10 '22 16:05 frankliee

Likely. This PR will support creating v2 views.

jzhuge avatar May 10 '22 16:05 jzhuge

Likely. This PR will support creating v2 views.

Thanks. What about other commands, such as dropview, alterivew? They also do not support v2 view, currently.

frankliee avatar May 11 '22 08:05 frankliee

Yes, many view DDL commands will be implemented, you can find a complete list in SPIP.

jzhuge avatar May 11 '22 15:05 jzhuge

Thanks @ljfgem @wmoustafa for the comments. Looking ...

Also may add a few unit tests to illustrate some scenarios.

jzhuge avatar Aug 29 '22 20:08 jzhuge

Hi Melin,

I opened a new PR because #28147 https://github.com/apache/spark/pull/28147 was closed and it was way out of sync from the current main branch.

Thanks,

On Fri, Mar 11, 2022 at 1:02 AM melin @.***> wrote:

#28147 https://github.com/apache/spark/pull/28147 PR available? Why submit a new PR?

— Reply to this email directly, view it on GitHub https://github.com/apache/spark/pull/35636#issuecomment-1064910166, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOL5JCGJGNK3D4FROQQWBTU7MDZLANCNFSM5PFSWFAQ . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

You are receiving this because you were mentioned.Message ID: @.***>

-- John Zhuge

jzhuge avatar Oct 11 '22 07:10 jzhuge

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar May 06 '23 00:05 github-actions[bot]

@jzhuge Have a plan to complete this PR?

melin avatar Oct 13 '23 04:10 melin

Yes

On Thu, Oct 12, 2023 at 9:41 PM melin @.***> wrote:

@jzhuge https://github.com/jzhuge Have a plan to complete this PR?

— Reply to this email directly, view it on GitHub https://github.com/apache/spark/pull/35636#issuecomment-1760790245, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAOL5JE3Q65KVSAQLBHQMW3X7DA6ZANCNFSM5PFSWFAQ . You are receiving this because you were mentioned.Message ID: @.***>

jzhuge avatar Oct 13 '23 04:10 jzhuge