elementary icon indicating copy to clipboard operation
elementary copied to clipboard
verified publisher icon elementary-data

Error when building Elementary models in ClickHouse

Open tnhgiang opened this issue 10 months ago • 0 comments

Describe the bug There are some errors occuring in building Elementary

To Reproduce Steps to reproduce the behavior:

According to the tutorial

  1. Add elementary to packages.yml
  2. Add to your dbt_project.yml
  3. Import the package and build Elementary models
  4. See error

Expected behavior No errors should occur

Log

# dbt run --select elementary
08:37:19  Running with dbt=1.8.8
08:37:19  Registered adapter: clickhouse=1.8.7
08:37:19  Found 34 models, 2 operations, 2 sources, 1265 macros
08:37:19  
08:37:20  dbt_clickhouse adapter: Unexpected server exception dropping table
08:37:20  
08:37:20  Running 1 on-run-start hook
08:37:20  1 of 1 START hook: elementary.on-run-start.0 ................................... [RUN]
08:37:20  1 of 1 OK hook: elementary.on-run-start.0 ...................................... [OK in 0.00s]
08:37:20  
08:37:20  Concurrency: 1 threads (target='local')
08:37:20  
08:37:20  1 of 29 START sql incremental model `nyc_taxi_elementary`.`data_monitoring_metrics`  [RUN]
08:37:20  1 of 29 OK created sql incremental model `nyc_taxi_elementary`.`data_monitoring_metrics`  [OK in 0.11s]
08:37:20  2 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_columns` ........ [RUN]
08:37:20  2 of 29 ERROR creating sql incremental model `nyc_taxi_elementary`.`dbt_columns`  [ERROR in 0.20s]
08:37:20  3 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_exposures` ...... [RUN]
08:37:20  3 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_exposures` . [OK in 0.05s]
08:37:20  4 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_invocations` .... [RUN]
08:37:20  4 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_invocations`  [OK in 0.05s]
08:37:20  5 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_metrics` ........ [RUN]
08:37:20  5 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_metrics` ... [OK in 0.04s]
08:37:20  6 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_models` ......... [RUN]
08:37:20  6 of 29 ERROR creating sql incremental model `nyc_taxi_elementary`.`dbt_models`  [ERROR in 0.07s]
08:37:20  7 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_run_results` .... [RUN]
08:37:20  7 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_run_results`  [OK in 0.04s]
08:37:20  8 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_seeds` .......... [RUN]
08:37:21  8 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_seeds` ..... [OK in 0.04s]
08:37:21  9 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_snapshots` ...... [RUN]
08:37:21  9 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_snapshots` . [OK in 0.04s]
08:37:21  10 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_source_freshness_results`  [RUN]
08:37:21  10 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_source_freshness_results`  [OK in 0.03s]
08:37:21  11 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_sources` ....... [RUN]
08:37:21  11 of 29 ERROR creating sql incremental model `nyc_taxi_elementary`.`dbt_sources`  [ERROR in 0.06s]
08:37:21  12 of 29 START sql incremental model `nyc_taxi_elementary`.`dbt_tests` ......... [RUN]
08:37:21  12 of 29 OK created sql incremental model `nyc_taxi_elementary`.`dbt_tests` .... [OK in 0.05s]
08:37:21  13 of 29 START sql incremental model `nyc_taxi_elementary`.`elementary_test_results`  [RUN]
08:37:21  13 of 29 OK created sql incremental model `nyc_taxi_elementary`.`elementary_test_results`  [OK in 0.04s]
08:37:21  14 of 29 START sql table model `nyc_taxi_elementary`.`metadata` ................ [RUN]
08:37:21  14 of 29 OK created sql table model `nyc_taxi_elementary`.`metadata` ........... [OK in 0.04s]
08:37:21  15 of 29 START sql incremental model `nyc_taxi_elementary`.`schema_columns_snapshot`  [RUN]
08:37:21  15 of 29 OK created sql incremental model `nyc_taxi_elementary`.`schema_columns_snapshot`  [OK in 0.03s]
08:37:21  16 of 29 START sql view model `nyc_taxi_elementary`.`metrics_anomaly_score` .... [RUN]
08:37:21  16 of 29 ERROR creating sql view model `nyc_taxi_elementary`.`metrics_anomaly_score`  [ERROR in 0.04s]
08:37:21  17 of 29 START sql view model `nyc_taxi_elementary`.`monitors_runs` ............ [RUN]
08:37:21  17 of 29 OK created sql view model `nyc_taxi_elementary`.`monitors_runs` ....... [OK in 0.01s]
08:37:21  18 of 29 START sql view model `nyc_taxi_elementary`.`job_run_results` .......... [RUN]
08:37:21  18 of 29 OK created sql view model `nyc_taxi_elementary`.`job_run_results` ..... [OK in 0.02s]
08:37:21  19 of 29 SKIP relation nyc_taxi_elementary.model_run_results ................... [SKIP]
08:37:21  20 of 29 START sql view model `nyc_taxi_elementary`.`seed_run_results` ......... [RUN]
08:37:21  20 of 29 OK created sql view model `nyc_taxi_elementary`.`seed_run_results` .... [OK in 0.02s]
08:37:21  21 of 29 START sql view model `nyc_taxi_elementary`.`snapshot_run_results` ..... [RUN]
08:37:21  21 of 29 OK created sql view model `nyc_taxi_elementary`.`snapshot_run_results`  [OK in 0.01s]
08:37:21  22 of 29 SKIP relation nyc_taxi_elementary.alerts_dbt_source_freshness ......... [SKIP]
08:37:21  23 of 29 SKIP relation nyc_taxi_elementary.dbt_artifacts_hashes ................ [SKIP]
08:37:21  24 of 29 START sql view model `nyc_taxi_elementary`.`alerts_anomaly_detection` . [RUN]
08:37:21  24 of 29 OK created sql view model `nyc_taxi_elementary`.`alerts_anomaly_detection`  [OK in 0.02s]
08:37:21  25 of 29 START sql view model `nyc_taxi_elementary`.`alerts_dbt_tests` ......... [RUN]
08:37:21  25 of 29 OK created sql view model `nyc_taxi_elementary`.`alerts_dbt_tests` .... [OK in 0.01s]
08:37:21  26 of 29 START sql view model `nyc_taxi_elementary`.`alerts_schema_changes` .... [RUN]
08:37:23  26 of 29 OK created sql view model `nyc_taxi_elementary`.`alerts_schema_changes`  [OK in 0.02s]
08:37:23  27 of 29 START sql incremental model `nyc_taxi_elementary`.`test_result_rows` .. [RUN]
08:37:23  27 of 29 OK created sql incremental model `nyc_taxi_elementary`.`test_result_rows`  [OK in 0.04s]
08:37:23  28 of 29 SKIP relation nyc_taxi_elementary.anomaly_threshold_sensitivity ....... [SKIP]
08:37:23  29 of 29 SKIP relation nyc_taxi_elementary.alerts_dbt_models ................... [SKIP]
08:37:23  
08:37:23  Running 1 on-run-end hook
08:37:23  dbt_clickhouse adapter: Unexpected server exception dropping table
08:37:24  Database error while running on-run-end
08:37:24  
08:37:24  Finished running 15 incremental models, 1 table model, 13 view models, 1 project hook in 0 hours 0 minutes and 4.37 seconds (4.37s).
08:37:24  
08:37:24  Completed with 5 errors and 0 warnings:
08:37:24  
08:37:24    Database Error in model dbt_columns (models/edr/dbt_artifacts/dbt_columns.sql)
  HTTPDriver for http://clickhouse:8123 received ClickHouse error code 115
   Code: 115. DB::Exception: Unknown setting 'replicated_deduplication_window': for storage Memory. (UNKNOWN_SETTING) (version 24.8.12.28 (official build))
  compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_columns.sql
08:37:24  
08:37:24    Database Error in model dbt_models (models/edr/dbt_artifacts/dbt_models.sql)
  HTTPDriver for http://clickhouse:8123 received ClickHouse error code 115
   Code: 115. DB::Exception: Unknown setting 'replicated_deduplication_window': for storage Memory. (UNKNOWN_SETTING) (version 24.8.12.28 (official build))
  compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_models.sql
08:37:24  
08:37:24    Database Error in model dbt_sources (models/edr/dbt_artifacts/dbt_sources.sql)
  HTTPDriver for http://clickhouse:8123 received ClickHouse error code 115
   Code: 115. DB::Exception: Unknown setting 'replicated_deduplication_window': for storage Memory. (UNKNOWN_SETTING) (version 24.8.12.28 (official build))
  compiled code at target/run/elementary/models/edr/dbt_artifacts/dbt_sources.sql
08:37:24  
08:37:24    Database Error in model metrics_anomaly_score (models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql)
  HTTPDriver for http://clickhouse:8123 received ClickHouse error code 63
   Code: 63. DB::Exception: Aggregate function with name 'stddev' does not exist. In scope SELECT id, full_table_name, column_name, dimension, dimension_value, metric_name, metric_value, source_value, bucket_start, bucket_end, bucket_duration_hours, updated_at, avg(metric_value) OVER (PARTITION BY metric_name, full_table_name, column_name ORDER BY bucket_start ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS training_avg, stddev(metric_value) OVER (PARTITION BY metric_name, full_table_name, column_name ORDER BY bucket_start ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS training_stddev, count(metric_value) OVER (PARTITION BY metric_name, full_table_name, column_name ORDER BY bucket_start ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS training_set_size, anyLast(bucket_end) OVER (PARTITION BY metric_name, full_table_name, column_name ORDER BY bucket_start ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS training_end, any(bucket_end) OVER (PARTITION BY metric_name, full_table_name, column_name ORDER BY bucket_start ASC ROWS BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS training_start FROM data_monitoring_metrics GROUP BY 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12. (UNKNOWN_AGGREGATE_FUNCTION) (version 24.8.12.28 (official build))
  compiled code at target/run/elementary/models/edr/data_monitoring/anomaly_detection/metrics_anomaly_score.sql
08:37:24  
08:37:24    on-run-end failed, error:
 HTTPDriver for http://clickhouse:8123 received ClickHouse error code 47
 Code: 47. DB::Exception: Missing columns: 'current_timestamp' while processing query: 'CAST(current_timestamp, 'timestamp')', required columns: 'current_timestamp' 'current_timestamp': While executing ValuesBlockInputFormat. (UNKNOWN_IDENTIFIER) (version 24.8.12.28 (official build))
08:37:24  
08:37:24  Done. PASS=20 WARN=0 ERROR=5 SKIP=5 TOTAL=30

Environment (please complete the following information):

  • Elementary CLI (edr) version: I didn't install Elementary CLI
  • Elementary dbt package version: 0.16.1
  • dbt version: 1.8.8
  • Data warehouse: ClickHouse v24.8.12.28

tnhgiang avatar Jan 27 '25 08:01 tnhgiang