elementary icon indicating copy to clipboard operation
elementary copied to clipboard
verified publisher icon elementary-data

[bug] prevent too big payload in edr monitor that will be rejected by Teams

Open FrankTub opened this issue 1 month ago • 0 comments

Describe the bug

This issue is raised when you have configured some test samples too be included in the alert that is sent by edr monitor to Microsoft Teams and the resulting payload that is generated by Elementary is too big. So this will not be raised when you have like two columns and two failed rows. But this will typically be raised when you have like 25 rows with e.g. 10 columns.

Example of the error that you will get:

[2025-10-29, 11:37:54 UTC] {logging_mixin.py:190} INFO - Sending alert to team debtors using command: /usr/local/airflow/production_dbt_venv/bin/edr monitor --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --filters tags:debtors --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25 --report-url $REPORT_URL
[2025-10-29, 11:37:54 UTC] {custom_operator.py:87} INFO - Copying /usr/local/airflow/dags/dbt/production to /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752
[2025-10-29, 11:37:55 UTC] {custom_operator.py:108} INFO - Running command in project dir: /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752
[2025-10-29, 11:37:55 UTC] {custom_operator.py:109} INFO - /usr/local/airflow/production_dbt_venv/bin/edr monitor --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --filters tags:debtors --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25 --report-url $REPORT_URL
[2025-10-29, 11:37:58 UTC] {logging_mixin.py:190} WARNING - /usr/local/airflow/.local/lib/python3.11/site-packages/watchtower/__init__.py:430 WatchtowerWarning: Received empty message. Empty messages cannot be sent to CloudWatch Logs
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO -     ________                          __
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO -    / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO -   / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO -  / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ /
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO -                                                  /____/
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
[2025-10-29, 11:37:58 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:37:58 — INFO — Running with edr=0.20.0
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — edr (0.20.0) and Elementary's dbt package (0.20.1) are compatible.
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — Elementary's database and schema: '"analytics.elementary"'
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — Running internal dbt run to populate alerts
[2025-10-29, 11:38:17 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:17 — INFO — Running dbt command run -s elementary_cli.alerts.alerts_v2 --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default --vars {"days_back": 1}
[2025-10-29, 11:38:27 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:27 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:34 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:34 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:36 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:36 — INFO — Update skipped alerts
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:37 — ERROR — Could not send the alert - <class 'elementary.monitor.alerts.test_alert.TestAlertModel'>.
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:37 — ERROR — Could not send the alert - 49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54. Full alert: {"id": "49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "alert_class_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "model_unique_id": "model.beequip_etl_dbt.states_on_contracts", "detected_at": "2025-10-29 11:30:14 UTC", "database_name": "analytics", "schema_name": "validations", "owners": ["Some Owner"], "tags": ["data-quality", "debtors"], "subscribers": [], "status": "fail", "suppression_interval": 0, "test_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "elementary_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "test_name": "assert_inactive_contracts_in_correct_state", "test_display_name": "Assert Inactive Contracts In Correct State", "severity": "ERROR", "table_name": "states_on_contracts", "table_full_name": "analytics.validations.states_on_contracts", "test_type": "dbt_test", "test_sub_type": "generic", "test_sub_type_display_name": "Generic", "test_results_description": "Got 193 results, configured to fail if != 0", "test_results_query": "select\n    *\nfrom (select * from analytics.validations.states_on_contracts where is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)) dbt_subquery\n\nwhere not(state in ('early_buyout_in_full', 'paid_in_full', 'shortfall'))", "test_short_name": "assert_inactive_contracts_in_correct_state", "test_description": "Indien een contract in Soft4Lease inactief is volgens het amortisatieschema dan moet de status aangepast worden. Pas dit in Soft4Lease aan!", "other": {}, "test_params": {"expression": "state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')", "model": "{{ get_where_subquery(ref('states_on_contracts')) }}"}, "test_rows_sample": [$MY_TEST_RESULTS], "column_name": null, "env": null}
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - Sending alerts |████████████████████████████████████████| 1/1 [100%] in 0.7s (1.50/s)
[2025-10-29, 11:38:37 UTC] {custom_operator.py:117} INFO - 2025-10-29 11:38:37 — INFO — Update sent alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 9005
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 469
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.data_monitoring:edr (0.20.0) and Elementary's dbt package (0.20.1) are compatible.
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 445
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.data_monitoring:Elementary's database and schema: '"analytics.elementary"'
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Running internal dbt run to populate alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command run -s elementary_cli.alerts.alerts_v2 --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default --vars {"days_back": 1}
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_pending_alerts", "macro_args": {"days_back": 1, "type": null}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 688397
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.clients.dbt.command_line_dbt_runner:Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}} --project-dir /usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project --profiles-dir /tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752 --target default
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - DEBUG:elementary.clients.dbt.command_line_dbt_runner:Result bytes size for command '['--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_last_alert_sent_times", "macro_args": {"days_back": 1, "type": null}}', '--project-dir', '/usr/local/airflow/production_dbt_venv/lib/python3.11/site-packages/elementary/monitor/dbt_project', '--profiles-dir', '/tmp/production_data_quality_alerts.dq_alert_debtors_20251029113752', '--target', 'default']' is 3189
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.fetchers.alerts.alerts:Update skipped alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - on 0: ERROR:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Could not send the alert - <class 'elementary.monitor.alerts.test_alert.TestAlertModel'>.
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - on 1: ERROR:elementary.monitor.data_monitoring.alerts.data_monitoring_alerts:Could not send the alert - 49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54. Full alert: {"id": "49016812-8833-45e3-9184-428cc8bfcbf7.test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "alert_class_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "model_unique_id": "model.beequip_etl_dbt.states_on_contracts", "detected_at": "2025-10-29 11:30:14 UTC", "database_name": "analytics", "schema_name": "validations", "owners": ["Some Owner"], "tags": ["data-quality", "debtors"], "subscribers": [], "status": "fail", "suppression_interval": 0, "test_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54", "elementary_unique_id": "test.beequip_etl_dbt.assert_inactive_contracts_in_correct_state.0188634a54.None.generic", "test_name": "assert_inactive_contracts_in_correct_state", "test_display_name": "Assert Inactive Contracts In Correct State", "severity": "ERROR", "table_name": "states_on_contracts", "table_full_name": "analytics.validations.states_on_contracts", "test_type": "dbt_test", "test_sub_type": "generic", "test_sub_type_display_name": "Generic", "test_results_description": "Got 193 results, configured to fail if != 0", "test_results_query": "select\n    *\nfrom (select * from analytics.validations.states_on_contracts where is_active = false and date_trunc('month', last_payment_on) != date_trunc('month', current_date)) dbt_subquery\n\nwhere not(state in ('early_buyout_in_full', 'paid_in_full', 'shortfall'))", "test_short_name": "assert_inactive_contracts_in_correct_state", "test_description": "Indien een contract in Soft4Lease inactief is volgens het amortisatieschema dan moet de status aangepast worden. Pas dit in Soft4Lease aan!", "other": {}, "test_params": {"expression": "state in ('early_buyout_in_full', 'paid_in_full', 'shortfall')", "model": "{{ get_where_subquery(ref('states_on_contracts')) }}"}, "test_rows_sample": [$MY_TEST_RESULTS], "column_name": null, "env": null}
[2025-10-29, 11:38:38 UTC] {custom_operator.py:120} WARNING - INFO:elementary.monitor.fetchers.alerts.alerts:Update sent alerts
[2025-10-29, 11:38:38 UTC] {custom_operator.py:133} ERROR - Command failed with exit code 1

To Reproduce Steps to reproduce the behavior:

The dbt_project.yml:

vars:
  test_sample_row_count: 1000
  1. Create a model with a test that you know will fail with like 10 columns or so. Make sure that there are lot of rows that will fail.
  2. run command like `edr monitor --teams-webhook $TEAMS_WEBHOOK --profile-target default --days-back 1 --maximum-columns-in-alert-samples 25
  3. Get an error

Expected behavior

All alerts are sent to teams without any issues. I think if a payload get's too big some of the data should be omitted or something. My suggestion would be if a payload is above a threshold the test samples is skipped and another message is displayed instead.

Screenshots N/A

Environment (please complete the following information):

  • Elementary CLI (edr) version: 0.20.0
  • Elementary dbt package version: 0.20.1
  • dbt version you're using: 1.9.4
  • Data warehouse: Snowflake
  • Infrastructure details: Production using Airflow

Additional context Add any other context about the problem here.

Would you be willing to contribute a fix for this issue?

Not sure where to begin

FrankTub avatar Nov 04 '25 08:11 FrankTub