elementary icon indicating copy to clipboard operation
elementary copied to clipboard

Published 20 hours ago • verified publisher icon elementary-data

Running DBT Test on DB with elementary-schema foreign wrapper

Open nurdin-i opened this issue 1 year ago • 1 comments

Describe the bug When running DBT Tests on a DB that is using a foreign wrapper and elementary schema from a different database. It acts like the elementary schema and its tables do not exist while they do.

When running the DBT test:

15:03:49 Running 1 on-run-end hook 15:03:49 Elementary: Unable to insert rows to non-existent table. Please run "dbt run -s elementary". 15:03:49 1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN] 15:03:49 1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s]

When running dbt run -s elementary on the SAME DB (that already has a foreign wrapper and FOREIGN elementary schema imported from a different DB):

15:09:23 Concurrency: 4 threads (target='new_database') 15:09:23 15:09:23 1 of 29 START sql incremental model elementary.data_monitoring_metrics ......... [RUN] 15:09:23 2 of 29 START sql incremental model elementary.dbt_exposures ................... [RUN] 15:09:23 3 of 29 START sql incremental model elementary.dbt_invocations ................. [RUN] 15:09:23 4 of 29 START sql incremental model elementary.dbt_metrics ..................... [RUN] 15:09:24 3 of 29 ERROR creating sql incremental model elementary.dbt_invocations ........ [ERROR in 1.43s] 15:09:24 5 of 29 START sql incremental model elementary.dbt_models ...................... [RUN] 15:09:24 1 of 29 ERROR creating sql incremental model elementary.data_monitoring_metrics [ERROR in 1.46s] 15:09:24 6 of 29 START sql incremental model elementary.dbt_run_results ................. [RUN] 15:09:24 2 of 29 ERROR creating sql incremental model elementary.dbt_exposures .......... [ERROR in 1.49s] 15:09:24 7 of 29 START sql incremental model elementary.dbt_seeds ....................... [RUN] 15:09:24 4 of 29 ERROR creating sql incremental model elementary.dbt_metrics ............ [ERROR in 1.55s] 15:09:24 8 of 29 START sql incremental model elementary.dbt_snapshots ................... [RUN] 15:09:26 6 of 29 ERROR creating sql incremental model elementary.dbt_run_results ........ [ERROR in 1.27s] 15:09:26 9 of 29 START sql incremental model elementary.dbt_source_freshness_results .... [RUN] 15:09:26 5 of 29 ERROR creating sql incremental model elementary.dbt_models ............. [ERROR in 1.35s] 15:09:26 10 of 29 START sql incremental model elementary.dbt_sources .................... [RUN] 15:09:26 7 of 29 ERROR creating sql incremental model elementary.dbt_seeds .............. [ERROR in 1.34s] 15:09:26 11 of 29 START sql incremental model elementary.dbt_tests ...................... [RUN] 15:09:26 8 of 29 ERROR creating sql incremental model elementary.dbt_snapshots .......... [ERROR in 1.39s] 15:09:26 12 of 29 START sql incremental model elementary.elementary_test_results ........ [RUN] 15:09:27 9 of 29 ERROR creating sql incremental model elementary.dbt_source_freshness_results [ERROR in 1.29s] 15:09:27 13 of 29 START sql view model elementary.filtered_information_schema_columns ... [RUN] 15:09:27 10 of 29 ERROR creating sql incremental model elementary.dbt_sources ........... [ERROR in 1.28s] 15:09:27 14 of 29 START sql view model elementary.filtered_information_schema_tables .... [RUN] 15:09:27 11 of 29 ERROR creating sql incremental model elementary.dbt_tests ............. [ERROR in 1.32s] 15:09:27 15 of 29 START sql table model elementary.metadata ............................. [RUN] 15:09:27 12 of 29 ERROR creating sql incremental model elementary.elementary_test_results [ERROR in 1.40s] 15:09:27 16 of 29 START sql incremental model elementary.schema_columns_snapshot ........ [RUN] 15:09:28 14 of 29 ERROR creating sql view model elementary.filtered_information_schema_tables [ERROR in 1.43s] 15:09:28 17 of 29 SKIP relation elementary.job_run_results .............................. [SKIP] 15:09:28 18 of 29 SKIP relation elementary.metrics_anomaly_score ........................ [SKIP] 15:09:28 19 of 29 SKIP relation elementary.monitors_runs ................................ [SKIP] 15:09:28 20 of 29 SKIP relation elementary.model_run_results ............................ [SKIP] 15:09:28 21 of 29 SKIP relation elementary.snapshot_run_results ......................... [SKIP] 15:09:28 22 of 29 SKIP relation elementary.alerts_dbt_source_freshness .................. [SKIP] 15:09:28 23 of 29 SKIP relation elementary.dbt_artifacts_hashes ......................... [SKIP] 15:09:28 24 of 29 SKIP relation elementary.alerts_anomaly_detection ..................... [SKIP] 15:09:28 25 of 29 SKIP relation elementary.alerts_dbt_tests ............................. [SKIP] 15:09:28 26 of 29 SKIP relation elementary.alerts_schema_changes ........................ [SKIP] 15:09:28 27 of 29 SKIP relation elementary.test_result_rows ............................. [SKIP] 15:09:28 28 of 29 SKIP relation elementary.anomaly_threshold_sensitivity ................ [SKIP] 15:09:28 29 of 29 SKIP relation elementary.alerts_dbt_models ............................ [SKIP] 15:09:29 15 of 29 ERROR creating sql table model elementary.metadata .................... [ERROR in 1.49s] 15:09:29 13 of 29 ERROR creating sql view model elementary.filtered_information_schema_columns [ERROR in 1.62s] 15:09:29 16 of 29 ERROR creating sql incremental model elementary.schema_columns_snapshot [ERROR in 1.37s] 15:09:30 15:09:30 Running 1 on-run-end hook 15:09:30 1 of 1 START hook: elementary.on-run-end.0 ..................................... [RUN] 15:09:30 1 of 1 OK hook: elementary.on-run-end.0 ........................................ [OK in 0.00s] 15:09:30 15:09:30 15:09:30 Finished running 14 incremental models, 14 view models, 1 table model, 2 hooks in 0 hours 0 minutes and 11.16 seconds (11.16s). 15:09:30 15:09:30 Completed with 16 errors and 0 warnings: 15:09:30 15:09:30 Database Error in model dbt_invocations (models\edr\dbt_artifacts\dbt_invocations.sql) 15:09:30 relation "dbt_invocations" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_invocations.sql 15:09:30 15:09:30 Database Error in model data_monitoring_metrics (models\edr\data_monitoring\data_monitoring\data_monitoring_metrics.sql) 15:09:30 relation "data_monitoring_metrics" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\data_monitoring\data_monitoring\data_monitoring_metrics.sql 15:09:30 15:09:30 Database Error in model dbt_exposures (models\edr\dbt_artifacts\dbt_exposures.sql) 15:09:30 relation "dbt_exposures" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_exposures.sql 15:09:30 15:09:30 Database Error in model dbt_metrics (models\edr\dbt_artifacts\dbt_metrics.sql) 15:09:30 relation "dbt_metrics" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_metrics.sql 15:09:30 15:09:30 Database Error in model dbt_run_results (models\edr\dbt_artifacts\dbt_run_results.sql) 15:09:30 relation "dbt_run_results" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_run_results.sql 15:09:30 15:09:30 Database Error in model dbt_models (models\edr\dbt_artifacts\dbt_models.sql) 15:09:30 relation "dbt_models" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_models.sql 15:09:30 15:09:30 Database Error in model dbt_seeds (models\edr\dbt_artifacts\dbt_seeds.sql) 15:09:30 relation "dbt_seeds" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_seeds.sql 15:09:30 15:09:30 Database Error in model dbt_snapshots (models\edr\dbt_artifacts\dbt_snapshots.sql) 15:09:30 relation "dbt_snapshots" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_snapshots.sql 15:09:30 15:09:30 Database Error in model dbt_source_freshness_results (models\edr\run_results\dbt_source_freshness_results.sql) 15:09:30 relation "dbt_source_freshness_results" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\run_results\dbt_source_freshness_results.sql 15:09:30 15:09:30 Database Error in model dbt_sources (models\edr\dbt_artifacts\dbt_sources.sql) 15:09:30 relation "dbt_sources" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_sources.sql 15:09:30 15:09:30 Database Error in model dbt_tests (models\edr\dbt_artifacts\dbt_tests.sql) 15:09:30 relation "dbt_tests" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\dbt_artifacts\dbt_tests.sql 15:09:30 15:09:30 Database Error in model elementary_test_results (models\edr\run_results\elementary_test_results.sql) 15:09:30 relation "elementary_test_results" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\run_results\elementary_test_results.sql 15:09:30 15:09:30 Database Error in model filtered_information_schema_tables (models\edr\metadata_store\filtered_information_schema_tables.sql) 15:09:30 relation "filtered_information_schema_tables" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\metadata_store\filtered_information_schema_tables.sql 15:09:30 15:09:30 Database Error in model metadata (models\edr\system\metadata.sql) 15:09:30 relation "metadata" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\system\metadata.sql 15:09:30 15:09:30 Database Error in model filtered_information_schema_columns (models\edr\metadata_store\filtered_information_schema_columns.sql) 15:09:30 relation "filtered_information_schema_columns" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\metadata_store\filtered_information_schema_columns.sql 15:09:30 15:09:30 Database Error in model schema_columns_snapshot (models\edr\data_monitoring\schema_changes\schema_columns_snapshot.sql) 15:09:30 relation "schema_columns_snapshot" already exists 15:09:30 compiled Code at ./target\run\elementary\models\edr\data_monitoring\schema_changes\schema_columns_snapshot.sql 15:09:30 15:09:30 Done. PASS=0 WARN=0 ERROR=16 SKIP=13 TOTAL=29

To Reproduce Steps to reproduce the behavior:

  1. Create elementary schema in DB 1.
  2. Create foreign wrapper - import elementary schema in DB 2 from DB 1 (from step 1.)
  3. Run DB Tests on DB 2 schema (that has foreign wrapper)
  4. ERROR encountered.

Expected behavior

Output results successfully inserted into elementary schema/tables.

Screenshots

PROOF that new_database schema has elementary schema imported (foreign):

image

Part of the code (full code ABOVE) when running dbt run -s elementary on DB that already has FOREIGN schema but doesn't want to output tests in it:

image

Environment (please complete the following information):

  • edr Version: 0.7.10
  • dbt version: 1.4.6

nurdin-i avatar May 18 '23 15:05 nurdin-i

Seems to be an issue in dbt get_telation. @nurdin-i reported to dbt.

https://github.com/dbt-labs/dbt-postgres/issues/53

Maayan-s avatar May 19 '23 08:05 Maayan-s

Hi all, Closing this issue since it had no activity in the past 3 months, but if it's still relevant for anyone please feel free to re-open.

Thanks, Itamar

haritamar avatar May 29 '24 01:05 haritamar