spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-52326][SQL] Add partitions related ExternalCatalogEvent and post them in corresponding operations

Open waterlx opened this issue 6 months ago • 1 comments

What changes were proposed in this pull request?

  1. Add Partitions related events, for the following external catalog operations: create / drop / alter / rename. A base "PartitionsEvent" is added by extending "TableEvent". One "PartitionsEvent" (and its subs) could contain the operation for one or more partitions. So it is named as "PartitionsEvent" in plural form, so are its subs.

  2. Post those events in the corresponding external catalog operations

Why are the changes needed?

The operation list extracted from Spark events are of great help for user to summarize what have been done against a db/table/function/partition, for the purpose of logging and/or auditing. In ExternalCatalogWithListener, there are events posted against db, table and function for all registered listeners. But an operation against operation does not have its event posted.

Does this PR introduce any user-facing change?

With this change, partition related operations are posted into register listener as events

How was this patch tested?

Enrich UT and also tested on local cluster

Was this patch authored or co-authored using generative AI tooling?

No

waterlx avatar May 27 '25 13:05 waterlx

@gatorsmile @dongjoon-hyun Can you help to review this PR? Thanks!

waterlx avatar May 27 '25 13:05 waterlx

We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable. If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!

github-actions[bot] avatar Sep 05 '25 00:09 github-actions[bot]