spark icon indicating copy to clipboard operation
spark copied to clipboard

[SPARK-52475][TESTS] Remove deprecated table.schema method from tests

Open szehon-ho opened this issue 6 months ago • 7 comments

What changes were proposed in this pull request?

Fix many compile warnings like

[warn] /Users/szehon.ho/repos/apache-spark/spark/sql/catalyst/src/test/scala/org/apache/spark/sql/connector/catalog/CatalogSuite.scala:282:18: method schema in trait Table is deprecated (since 3.4.0)
[warn] Applicable -Wconf / @nowarn filters for this warning: msg=<part of the message>, cat=deprecation, site=org.apache.spark.sql.connector.catalog.CatalogSuite, origin=org.apache.spark.sql.connector.catalog.Table.schema, version=3.4.0

Why are the changes needed?

Reduce compiler warnings

Does this PR introduce any user-facing change?

No

How was this patch tested?

Test only

Was this patch authored or co-authored using generative AI tooling?

No

szehon-ho avatar Jun 13 '25 03:06 szehon-ho

Here are a huge number of compilation warnings related to "method schema in trait Table is deprecated". Can we fix them all at once? Or, could we resolve all of them in tests in one go? Fixing them one by one would lead to a multitude of prs.

LuciferYang avatar Jun 13 '25 05:06 LuciferYang

And I think it's worth creating a JIRA ticket for this.

LuciferYang avatar Jun 13 '25 09:06 LuciferYang

Yea its a large number and a bit time consuming, so I was thinking to at least make progress and avoid conflict risk by reasonable size batches. Let me look at removing from all the tests

szehon-ho avatar Jun 13 '25 23:06 szehon-ho

Updated to remove from all tests and created a JIRA

szehon-ho avatar Jun 14 '25 02:06 szehon-ho

@szehon-ho Thank you for your work. However, it seems there are some compilation errors that need to be fixed first.

LuciferYang avatar Jun 14 '25 12:06 LuciferYang

sql-others has test failures:

  • org.apache.spark.sql.execution.datasources.v2.V2SessionCatalogTableSuite

In addition,

./build/mvn scalafmt:format -Dscalafmt.skip=false -Dscalafmt.validateOnly=false -Dscalafmt.changedOnly=false -pl sql/api -pl sql/connect/common -pl sql/connect/server -pl sql/connect/client/jvm

should be executed. @szehon-ho

LuciferYang avatar Jun 15 '25 09:06 LuciferYang

Sorry there's a lot of tests and I missed running some locally first, thanks for your help

szehon-ho avatar Jun 15 '25 19:06 szehon-ho

Merged to master for Apache Spark 4.1.0.

dongjoon-hyun avatar Jun 16 '25 14:06 dongjoon-hyun

late LGTM, thanks @szehon-ho

LuciferYang avatar Jun 17 '25 03:06 LuciferYang