elementary icon indicating copy to clipboard operation
elementary copied to clipboard

Data test broken when switching to materialized views on postgres

Open mateusz opened this issue 1 year ago • 0 comments

Describe the bug

On postgres, when using elementary.column_anomalies test on a column that is part of a table using materialized views, an "Unable to find column" error occurs during the elemenatary data test run.

To Reproduce

Create a table using materialized view materialization:

{{ config(materialized='materialized_view') }}

SELECT bill_usd FROM table

Configure test in DBT table spec:

models:
  - name: table
    columns:
      - name: bill_usd
        tests:
          - elementary.column_anomalies:
              detection_period:
                period: day
                count: 2
              training_period:
                period: day
                count: 35
              detection_delay:
                period: day
                count: 5
              column_anomalies:
                - sum
              config:
                severity: warn

Run elementary:

2024-08-27 12:05:46,796 | INFO | root | 00:05:42    Compilation Error in test ...bill_usd__day__5__day__2__day__35 (models/....yml)
2024-08-27 12:05:46,796 | INFO | root |   Unable to find column `bill_usd` in `....TABLE`
2024-08-27 12:05:46,796 | INFO | root |   
2024-08-27 12:05:46,796 | INFO | root |   > in macro test_column_anomalies (macros/edr/tests/test_column_anomalies.sql)
2024-08-27 12:05:46,797 | INFO | root |   > called by test ...bill_usd__day__5__day__2__day__35 (...)

This problem goes away when I remove the materialization, or switch to table materialization.

Expected behavior

I expect data tests to continue working when I switch to materialized_view materialization.

Screenshots

n/a

Environment (please complete the following information):

  • Elementary CLI (edr) version: 0.16.1
  • Elementary dbt package version: 0.16.0
  • dbt version you're using: 1.8.5
  • Data warehouse: postgres
  • Infrastructure details (e.g. operating system, prod / dev / staging, deployment infra, CI system, etc): freshly built docker image prefecthq/prefect:2-python3.9

mateusz avatar Aug 27 '24 00:08 mateusz