elementary
elementary copied to clipboard
Published
20 hours ago •
elementary-data
elementary-data
ERROR — Could not generate the report - Error: Failed to run dbt command
Describe the bug When below command is run
edr send-report --s3-bucket-name abc --bucket-file-path /edr/index.html --project-dir /opt/airflow/dbt-dir/ --profiles-dir /opt/airflow/dbt-dir/ -c /mnt/working-dir -st *** -ch elem_alerts
It throws below error
[2023-11-07, 21:10:29 EST] {subprocess.py:74} INFO - Running command: ['bash', '-c', 'edr send-report --s3-bucket-name abc --bucket-file-path /edr/index.html --project-dir /opt/airflow/dbt-dir/ --profiles-dir /opt/airflow/dbt-dir/ -c /mnt/working-dir -st *** -ch elem_alerts']
[2023-11-07, 21:10:29 EST] {subprocess.py:85} INFO - Output:
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - ________ __
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - / ____/ /__ ____ ___ ___ ____ / /_____ ________ __
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - / /___/ / __/ / / / / / __/ / / / /_/ /_/ / / / /_/ /
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - /_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/ \__, /
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - /____/
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO -
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO -
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:32 — INFO — Running with edr=0.12.0
[2023-11-07, 21:10:32 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:32 — INFO — Running dbt --log-format json deps --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Running with dbt=1.7.1
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Updating lock file in file path: /mnt/working-dir/edr/elementary/monitor/dbt_project/package-lock.yml
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Installing dbt-labs/dbt_utils
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Installed from version 0.8.6
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Updated version available: 1.1.1
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Installing elementary-data/elementary
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Installed from version 0.12.0
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Updated version available: 0.12.1
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO —
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - 2023-11-08 02:10:39 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']
[2023-11-07, 21:10:39 EST] {subprocess.py:92} INFO - Update your versions in packages.yml, then run dbt deps
[2023-11-07, 21:11:08 EST] {subprocess.py:92} INFO - 2023-11-08 02:11:08 — INFO — edr (0.12.0) and Elementary's dbt package (0.12.1) are compatible.
[2023-11-07, 21:11:17 EST] {subprocess.py:92} INFO - 2023-11-08 02:11:17 — INFO — Elementary's database and schema: '"DEV.METRICS_elementary"'
[2023-11-07, 21:11:17 EST] {subprocess.py:92} INFO - 2023-11-08 02:11:17 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /mnt/working-dir/edr/elementary/monitor/dbt_project --profiles-dir /opt/airflow/dbt-dir/
[2023-11-07, 21:11:34 EST] {subprocess.py:92} INFO - [31;20m2023-11-08 02:11:34 — ERROR — Could not generate the report - Error: Failed to run dbt command.
[2023-11-07, 21:11:34 EST] {subprocess.py:92} INFO - Please reach out to our community for help with this issue.[0m
[2023-11-07, 21:11:34 EST] {subprocess.py:92} INFO - NoneType: None
[2023-11-07, 21:11:35 EST] {subprocess.py:96} INFO - Command exited with return code 1
[2023-11-07, 21:11:35 EST] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
File "/mnt/working-dir/avalara_ds/airflow/operators/dbt_bash.py", line 97, in execute
super().execute(context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/operators/bash.py", line 194, in execute
raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
Expected behavior Should be able to run edr send-report command It was running fine before edr version 0.12.0
I also updated the dbt packages latest on 0.12.1 and run dbt deps and dbt run --select elementary
Environment (please complete the following information):
edr=0.12.0
dbt=1.7.1
packages:
## Docs: https://docs.elementary-data.com
## Monitor your data quality, operation and performance directly from your dbt project
- package: elementary-data/elementary
version: 0.12.1
## contains macros that can be (re)used across dbt projects
- package: dbt-labs/dbt_utils
version: 1.1.1
## helps assert expectations for different dataset characteristics and column values
- package: calogica/dbt_expectations
version: 0.10.0
## Macros that generate dbt code, and log it to the command line
- package: dbt-labs/codegen
version: 0.11.0
Additional context
Hello,
I recently encountered an issue while trying to generate a report using elementary-data. This occurred after updating dbt from version 1.6.4 to 1.7.2. I am currently using elementary-data version 0.12.1 on a Mac Mini M2 Pro.
The error appears when executing the edr report command in a VSCode terminal. Here is the output I received:
vscode ➜ /workspaces/dbt (main) $ poetry run edr report
________ __
/ ____/ /__ ____ ___ ___ ____ / /_____ ________ __
/ __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
/ /___/ / __/ / / / / / __/ / / / /_/ /_/ / / / /_/ /
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/ \__, /
/____/
Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary
2023-11-29 04:49:09 — INFO — Running with edr=0.12.0
2023-11-29 04:49:13 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Encountered an error:
Compilation Error
expected token 'end of print statement', got 'string'
line 15
{{ return "" }}
2023-11-29 04:49:20 — ERROR — Failed to parse Elementary's database and schema.
2023-11-29 04:49:20 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
2023-11-29 04:49:24 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error:
Compilation Error
expected token 'end of print statement', got 'string'
line 15
{{ return "" }}
Please reach out to our community for help with this issue.
NoneType: None
I downgraded [email protected], still encountered same error.
2023-11-29 05:12:40 — INFO — Running with edr=0.12.0
2023-11-29 05:12:40 — DEBUG — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_latest_invocation", "macro_args": {}} --project-dir /workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
[31;20m2023-11-29 05:12:44 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Encountered an error:
Compilation Error
expected token 'end of print statement', got 'string'
line 15
{{ return "" }}[0m
2023-11-29 05:12:44 — DEBUG — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}} --project-dir /workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
2023-11-29 05:12:47 — DEBUG — Could not get warehouse info.
Traceback (most recent call last):
File "/workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/clients/dbt/dbt_runner.py", line 82, in _run_command
result = subprocess.run(
^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.11/subprocess.py", line 571, in run
raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['dbt', '--log-format', 'json', 'run-operation', 'elementary.log_macro_results', '--args', '{"macro_name": "elementary_cli.get_adapter_type_and_unique_id", "macro_args": {}}', '--project-dir', '/workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/dbt_project']' returned non-zero exit status 2.
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/data_monitoring/data_monitoring.py", line 179, in _get_warehouse_info
self.internal_dbt_runner.run_operation(
File "/workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/clients/dbt/dbt_runner.py", line 154, in run_operation
success, command_output = self._run_command(
^^^^^^^^^^^^^^^^^^
File "/workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/clients/dbt/dbt_runner.py", line 93, in _run_command
raise DbtCommandError(err, command_args, logs=logs)
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Encountered an error:
Compilation Error
expected token 'end of print statement', got 'string'
line 15
{{ return "" }}
2023-11-29 05:12:47 — DEBUG — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_elementary_database_and_schema", "macro_args": {}} --project-dir /workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
[31;20m2023-11-29 05:12:51 — ERROR — Failed to parse Elementary's database and schema.[0m
2023-11-29 05:12:51 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /workspaces/dbt/.venv/lib/python3.11/site-packages/elementary/monitor/dbt_project
[31;20m2023-11-29 05:12:54 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Encountered an error:
Compilation Error
expected token 'end of print statement', got 'string'
line 15
{{ return "" }}
Please reach out to our community for help with this issue.[0m
NoneType: None
Hey @EFEXP @francescomucio , Does this error also occur if you run the command in a regular terminal and not VScode?
@Maayan-s I'm sorry, but we haven't tried running it outside of VS Code Devcontainer. We specifically need to execute it within this environment. We're using mcr.microsoft.com/devcontainers/python:3.10-bookworm and /bin/bash. Additionally, this error persists with edr version 0.13.2 and the dbt package version 0.13.0.
Is there any possible solution for this bug? For now, I found a workaround on the Slack channel: https://elementary-community.slack.com/archives/C02CTC89LAX/p1698699915707659?thread_ts=1697659818.900659&cid=C02CTC89LAX
Hey all, I had same error: Failed to run dbt command.
In our case, we run in MWAA, and our container image was out of memory. Have you tried to print the dbt.log? We got that info from there.
Hi all, Closing this issue since it had no activity in the past 3 months, but if it's still relevant for anyone please feel free to re-open.
Thanks, Itamar
