dbt-core icon indicating copy to clipboard operation
dbt-core copied to clipboard

[Feature] freshness tests on external tables should fail to parse `loaded_at_field`

Open dataders opened this issue 10 months ago • 1 comments

Is this your first time submitting a feature request?

  • [X] I have read the expectations for open source contributors
  • [X] I have searched the existing issues, and I could not find an existing issue for this feature
  • [X] I am requesting a straightforward extension of existing dbt functionality, rather than a Big Idea better suited to a discussion

Describe the feature

see https://github.com/dbt-labs/dbt-redshift/issues/757

metadata-generated freshness tests (ie those defined without a loaded_at_field) are not supported for External Tables. In addition to documenting this, dbt should error in this scenario and tell the user something more meaningful than current state.

Current behavior

Got an invalid result from "get_relation_last_modified" macro: []

Proposed Behavior

external table source XXX.XXX has metadata-based freshness test defined which is not supported. please add loaded_at_field or remove the freshness test

Describe alternatives you've considered

Perhaps some data platforms do support metadata-based freshness checks for external tables?

If so, perhaps this justifies an expansion of the Capability.TableLastModifiedMetadata entry of the Capability matrix (e.g. SnowflakeAdapter's )

Who will this benefit?

dbt users with external tables defined

Are you interested in contributing this feature?

No response

Anything else?

No response

dataders avatar Apr 18 '24 17:04 dataders

somewhat related to #9320?

dataders avatar Apr 30 '24 13:04 dataders