[Feature Request]Add a check to verify the required Spark confs

[zsxwing]

Feature request

Overview

We saw a few reported issues due to missing configs. We can add a check in DeltaLog to verify the following Spark confs.

--conf "spark.sql.extensions=io.delta.sql.DeltaSparkSessionExtension" --conf "spark.sql.catalog.spark_catalog=org.apache.spark.sql.delta.catalog.DeltaCatalog"

If these are not set, we can throw a user friendly error.

Motivation

Provide a user friendly error rather than throwing random weird errors.

Willingness to contribute

The Delta Lake Community encourages new feature contributions. Would you or another member of your organization be willing to contribute an implementation of this feature?

• [ ] Yes. I can contribute this feature independently.
• [ ] Yes. I would be willing to contribute this feature with guidance from the Delta Lake community.
• [x] No. I cannot contribute this feature at this time.

[ganeshchand]

@zsxwing I can work on this feature. Please feel free to assign this to me.

[scottsand-db]

Hi @ganeshchand - any update?

[ganeshchand]

It looks like https://github.com/delta-io/delta/pull/1238 is meant for throwing a prescriptive user friendly error if the required spark configs are not provided. Does that make this issue redundant?

[scottsand-db]

@ganeshchand - AFAIK, that error you referenced above is only thrown during Analysis. What this issue is asking is to add an explicit check in the DeltaLog, e.d. during initialization, that checks for the existence of the required spark confs.

[zsxwing]

In addition, if DeltaSparkSessionExtension is not set, nothing will be caught.

[ganeshchand]

Thanks for the clarification. I am working on it and will send the PR soon.

[ganeshchand]

@zsxwing I have pushed my changes here. Before raising the PR, I wanted to get feedback on whether I am throwing the right exception.

[zsxwing]

@ganeshchand could you open a PR so that we can discuss the details?