invoke Spark catalog plugin API to create tables

[cloud-fan]

Which Delta project/connector is this regarding?

• [x] Spark
• [ ] Standalone
• [ ] Flink
• [ ] Kernel
• [ ] Other (fill in here)

Description

Today, DeltaCatalog takes care of the Spark CREATE TABLE command and calls CreateDeltaTableCommand#run at the end. Within this command, spark.sessionState.catalog.createTable is called, which bypasses any custom catalog that overwrites spark_catalog. DeltaCatalog always creates tables in the Hive Metastore.

This PR fixes this issue by calling the Spark catalog plugin API to create the table, to respect custom catalog that overwrites spark_catalog, such as Unity Catalog.

How was this patch tested?

Locally tested it with Spark + Unity Catalog.

Does this PR introduce any user-facing changes?

no

[cloud-fan]

cc @tdas

[cloud-fan]

Option.when is available after Scala 2.13

[scovich]

Option.when is available after Scala 2.13

It's also available in 2.12 by importing the compat shim in https://github.com/delta-io/delta/blob/master/spark/src/main/scala/org/apache/spark/sql/util/ScalaExtensions.scala#L36

[tdas]

spark master failures seems un related.

[cloud-fan]

The master branch failure seems to be caused by the recent Spark refactors for Spark Connect. cc @hvanhovell

[tdas]

There seems to be other spark test failures.

[tdas]

@cloud-fan can you fix them?

[tdas]

merged. can you make a PR for branch 3.2?

[cloud-fan]

the 3.2 PR: https://github.com/delta-io/delta/pull/3603