dbt-databricks icon indicating copy to clipboard operation
dbt-databricks copied to clipboard

Runtime error with contracts when specifying struct as data_type

Open ferdyh opened this issue 1 year ago • 3 comments
trafficstars

Describe the bug

When I specify a column with data_type struct when using model contracts, then i get a runtime error.

Steps To Reproduce

  1. Created a model that has a struct column.
  2. Added a .yml file for the model, and added contracting
  3. Set datatype to struct
  4. dbt build that model

Expected behavior

Expected to accept the struct datatype without giving errors.

Screenshots and log output

10:13:42 Runtime Error in model [model_name] (models\[model_name].sql) struct (of class java.lang.String)

System information

The output of dbt --version:

Core:
  - installed: 1.8.3
  - latest:    1.8.3 - Up to date!

Plugins:
  - databricks: 1.8.3 - Up to date!
  - spark:      1.8.0 - Up to date!

Also broken in 1.7.13 and 1.7.17

The operating system you're using: Windows 11

The output of python --version: Python 3.11.9

ferdyh avatar Jul 01 '24 10:07 ferdyh

It looks like the query generates an cast(null as struct) as [column_name] whcih is invalid. WHen i define the whole struct type, it works.

ferdyh avatar Jul 01 '24 13:07 ferdyh

Thanks for the report. If you'd like to work on a PR to address, I'd be happy to help; otherwise due to resource constraints, this bug will probably not be fixed until the next time we work on improving column-specific feature support, hopefully later this year.

benc-db avatar Jul 01 '24 15:07 benc-db

There is an existing implementation for BigQuery, it'll be nice if nested columns will look the same for a different providers.

TBoris avatar Jul 03 '24 16:07 TBoris

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please remove the stale label or comment on the issue.

github-actions[bot] avatar Dec 31 '24 02:12 github-actions[bot]

Closing in favor of #897

benc-db avatar Jan 08 '25 20:01 benc-db