Randomly getting 'at least one column must be specified for the table'

[benc-db]

Describe the bug

When deploying materializations I will randomly get:

Runtime Error in <redacted>
20:40:59    Error running query: org.apache.spark.sql.AnalysisException: org.apache.hadoop.hive.ql.metadata.HiveException: at least one column must be specified for the table

This causes that step to fail.

Steps To Reproduce

I wish I knew. It doesn't happen consistently for any particular materialization. The crazy thing is that it happens sometimes for static materializations, like seeds.

Expected behavior

Not to randomly fail.

Screenshots and log output

See above

System information

The output of dbt --version:

Core:
  - installed: 1.1.0
  - latest:    1.2.0 - Update available!

  Your version of dbt-core is out of date!
  You can find instructions for upgrading here:
  https://docs.getdbt.com/docs/installation

Plugins:
  - databricks: 1.1.1 - Up to date!
  - spark:      1.1.0 - Update available!

The operating system you're using: macOS Big Sur 11.6.7

The output of python --version: Python 3.9.10

[allisonwang-db]

Hi @benc-db thanks for reporting the issue. Are you using Databricks clusters or SQL endpoints? If you are using clusters, which is the DBR version of your cluster?

[benc-db]

I believe I've seen it happen with both. Right now using an 11.1 cluster.

[nascimentocrafael]

Hi all,

Any update related on this issue? I am getting the same while trying to load a delta table using a Databricks cluster.

I am using the following version of Databricks Runtime: 10.4 LTS (includes Apache Spark 3.2.1, Scala 2.12) I am not using photon or any other specific configuration on the cluster.

[bilalaslamseattle]

@ueshin shouldn't this be fixed after your fix for tmp relation?

[ueshin]

@bilalaslamseattle No, this is not related to the tmp relation fix. cc @allisonwang-db

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.