elementary icon indicating copy to clipboard operation
elementary copied to clipboard
verified publisher icon elementary-data

Upgrading version 0.15.1 → 0.16.0 cause "Runtime Error: Could not find profile named 'elementary"

Open phalanx-hk opened this issue 1 year ago • 16 comments

Describe the bug Yesterday I upgrading elementary-data[bigquery] from 0.15.1 to 0.16.0, but Upgrading caused a Runtime Error. I include below the contents of the files and logs that are considered necessary for troubleshooting.

To Reproduce Steps to reproduce the behavior:

  1. setting dependencies and relevant yaml file as below pyproject.toml
"dbt-core >=1.8,<1.9",
"dbt-bigquery >=1.8,<1.9",
"elementary-data[bigquery] ==0.16.0",

profiles.yml

elementary:
  target: prod
  outputs:
    prod:
      dataset: dbt_elementary
      job_execution_timeout_seconds: 300
      job_retries: 1
      location: asia-northeast1
      method: oauth
      priority: interactive
      project: {project_id}
      threads: 4
      type: bigquery

packages.yml

packages:
  - package: elementary-data/elementary
    version: 0.16.0

dbt_project.yml

models:
  ...
  elementary:
  +schema: "dbt_elementary"
  +docs:
    show: false
flags:
  # Required from dbt 1.8 and above for certain Elementary features
  require_explicit_package_overrides_for_builtin_materializations: false
  1. run command
edr report --env prod --project-name {project_id}  --days-back=30
  1. see error
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-10 23:39:29 — INFO — Running with edr=0.16.0
2024-09-10 23:39:29 — INFO — Installing packages for edr internal dbt package...
2024-09-10 23:39:29 — INFO — Running dbt command --log-format json deps --project-dir {WORKING_DIR}/dbt/.venv/lib/python3.12/site-packages/elementary/monitor/dbt_project
{"data": {"exc": "Runtime Error\n  Could not find profile named 'elementary'"}, "info": {"category": "", "code": "Z002", "extra": {}, "invocation_id": "3ef899d9-d564-4170-a149-06641a6a5aac", "level": "error", "msg": "Encountered an error:\nRuntime Error\n  Could not find profile named 'elementary'", "name": "MainEncounteredError", "pid": 33465, "thread": "MainThread", "ts": "2024-09-10T14:39:31.495433Z"}}
2024-09-10 23:39:32 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Runtime Error
  Could not find profile named 'elementary'

Expected behavior A report of the dbt project will be generated.

Environment (please complete the following information):

  • Elementary CLI (edr) version: 0.16.0
  • Elementary dbt package version: 0.16.0
  • dbt version you're using: 1.8.6
  • Data warehouse: bigquery

phalanx-hk avatar Sep 11 '24 02:09 phalanx-hk

I have the exact same issue after upgrading from 0.15.1 to 0.16.1.

> dbt --version                                                                                                                                                         BI-811-dbt-sources-Fix-freshness-tests-and-alerting [3059a3f4] modified
Core:
  - installed: 1.8.6
  - latest:    1.8.6 - Up to date!

Plugins:
  - bigquery: 1.8.2 - Up to date!

Note that I also upgraded dbt from 1.7.x (dbt-core was 1.7.10 and dbt-bigquery was 1.7.6) to 1.8 at the same time.

geo909 avatar Sep 16 '24 09:09 geo909

Another tip, in case it helps.

I'm fixing the following:

# requirements.txt
dbt-core==1.8.6
dbt-bigquery==1.8.2

and

# packages.yml
packages
  - package: elementary-data/elementary
      version: 0.16.1
  1. With python package elementary-data==0.16.1 (same problem with 0.16.0):
> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:19:10 — INFO — Running with edr=0.16.1
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "c48be6d8-6481-490d-890d-0baec2937684", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:11.602498Z"}}
2024-09-17 10:19:12 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-17 10:19:14 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"'
2024-09-17 10:19:14 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "43151044-9715-4608-b6f1-a1fa68f06067", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 24886, "thread": "MainThread", "ts": "2024-09-17T07:19:15.226170Z"}}
2024-09-17 10:19:15 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 177, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 114, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
  1. With python package elementary-data==0.15.1:
> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                       main [0e491059] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.15.1, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-17 10:24:21 — INFO — Running with edr=0.15.1
2024-09-17 10:24:21 — INFO — edr internal dbt package is not up-to-date, updating it...
2024-09-17 10:24:21 — INFO — Running dbt --log-format json deps --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:24:25 — INFO — Running with dbt=1.8.6
2024-09-17 10:24:25 — INFO — Updating lock file in file path: /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project/package-lock.yml
2024-09-17 10:24:25 — INFO — Installing dbt-labs/dbt_utils
2024-09-17 10:24:25 — INFO — Installed from version 0.8.6
2024-09-17 10:24:25 — INFO — Updated version available: 1.3.0
2024-09-17 10:24:25 — INFO — Installing elementary-data/elementary
2024-09-17 10:24:25 — INFO — Installed from version 0.15.1
2024-09-17 10:24:25 — INFO — Updated version available: 0.16.1
2024-09-17 10:24:25 — INFO — 
2024-09-17 10:24:25 — INFO — Updates available for packages: ['dbt-labs/dbt_utils', 'elementary-data/elementary']                 
Update your versions in packages.yml, then run dbt deps
2024-09-17 10:24:44 — INFO — edr (0.15.1) and Elementary's dbt package (0.15.2) are compatible.
2024-09-17 10:24:49 — INFO — Elementary's database and schema: '"bigquery-xxxx.user_gtzanakis_dbt"'
2024-09-17 10:24:49 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:24:57 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:04 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:11 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:18 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:25 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:31 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:38 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:45 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:51 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-17 10:25:58 — INFO — Running dbt --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper

I don't know if it makes a difference but mentioning it just in case: I'm using pyenv virtualenv in ubuntu to handle my virtual environments, hopefully that's not the issue here.

geo909 avatar Sep 17 '24 07:09 geo909

Hey, @phalanx-hk , is your profiles.yml file located in your project directory or in the user directory?

  • if it is in the project directory, can you try adding --profiles-dir . to your command.

@geo909 from the logs you provided, i assume that in your profiles.yml file, i assume you define the following:

    keyfile: service-account-dbt.json

can you try setting the keyfile path to the absolute path of the file and see if this helps, something like

   keyfile: "/home/<user>/.../service-account-dbt.json

ofek1weiss avatar Sep 24 '24 14:09 ofek1weiss

@ofek1weiss Still doesn't work:

ferryhopper:
  target: dev_user
  outputs:
    dev_user:
      dataset: user_{{ env_var('DBT_USER') }}_dbt
      job_execution_timeout_seconds: 600
      job_retries: 1
      keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
      location: EU
      method: service-account
      priority: interactive
      project: bigquery-xxxxxx
      threads: 2
      type: bigquery

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"                                                                                                                                                                                                                                main [3a3a10e9] modified untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:49:36 — INFO — Running with edr=0.16.0
2024-09-24 17:49:36 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path 'ferryhopper' does not exist.
2024-09-24 17:49:36 — ERROR — Failed to parse Elementary's database and schema.
2024-09-24 17:49:36 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ferryhopper
2024-09-24 17:49:36 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path 'ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path 'ferryhopper' does not exist.

Tried this one too, just in case:

tzanakis@FHLT7FD7DL3:~/git/bi/dbt (*)
> edr report --project-dir "/home/tzanakis/git/bi/dbt/ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                                                     main [3a3a10e9] untracked
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-24 17:53:23 — INFO — Running with edr=0.16.0
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:25.257806Z"}}
2024-09-24 17:53:25 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
2024-09-24 17:53:28 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-24 17:53:28 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
{"data": {"exc": "Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'"}, "info": {"category": "", "code": "Q001", "extra": {}, "invocation_id": "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx", "level": "error", "msg": "Encountered an error while running operation: Database Error\n  [Errno 2] No such file or directory: 'service-account-dbt.json'", "name": "RunningOperationCaughtError", "pid": 226007, "thread": "MainThread", "ts": "2024-09-24T14:53:28.741599Z"}}
2024-09-24 17:53:29 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.

Note that, if I change the virtual environment to switch back to 0.15.1 and change absolutely nothing else, both the above commands run without errors.

geo909 avatar Sep 24 '24 15:09 geo909

@ofek1weiss Thanks for your feedback! I try adding --profiles-dir ., it works for me.

run command: edr report --env prod --project-name {project_id} --days-back=30 --profiles-dir .

phalanx-hk avatar Sep 25 '24 00:09 phalanx-hk

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

ofek1weiss avatar Sep 25 '24 05:09 ofek1weiss

@geo909 did you try both at the same time? (setting both the keyfile and the command flags to absolute paths)? or was it one at a time?

the reason i am asking is that in the logs for the second command, you can see the log No such file or directory: 'service-account-dbt.json', i would have expected to see No such file or directory: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"

Oops.. Apologies, that was my mistake. I was trying the full path in the wrong profile.

So, the following combination finally worked!

  • keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Played around with different expressions just in case, but relative paths expressed in different ways do not work, as expected probably. For example, the following:

  • keyfile: "./ferryhopper/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

geo909 avatar Sep 25 '24 07:09 geo909

@geo909 Firstly, that is great to hear :) I'll try to figure out what change caused the keyfile issue, and see if it is reversible. Regarding --profiles-dir, it should work with relative paths, so doing the following should work:

  • keyfile: "/home/tzanakis/git/bi/dbt/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper" can you please try that and let me know if it works/share logs if it does not? 🙏

ofek1weiss avatar Sep 25 '24 08:09 ofek1weiss

@ofek1weiss

No, the last combination does not work. To recap:

Works

  • keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "/home/tzanakis/git/bi/dbt/ferryhopper"                                                                                                                                                        main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:04:17 — INFO — Running with edr=0.16.0
2024-09-25 12:04:24 — INFO — edr (0.16.0) and Elementary's dbt package (0.16.1) are compatible.
2024-09-25 12:04:26 — INFO — Elementary's database and schema: '"bigquery-xxxxxx.user_gtzanakis_dbt"'
2024-09-25 12:04:26 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:32 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_source_freshness_results", "macro_args": {"days_back": 7, "invocations_per_test": 720}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:35 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:39 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_sources", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:42 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_exposures", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:46 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_singular_tests", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:49 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_runs", "macro_args": {"days_back": 7, "exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:54 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_dbt_models_test_coverage", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:04:58 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_nodes_depends_on_nodes", "macro_args": {"exclude_elementary": true}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:01 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocation", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper
2024-09-25 12:05:04 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_models_latest_invocations_data", "macro_args": {}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir /home/tzanakis/git/bi/dbt/ferryhopper

Does NOT work

  • keyfile: "/home/tzanakis/git/bi/dbt/ferryhopper/service-account-dbt.json"
  • edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"

Output:

> edr report --project-dir "ferryhopper" --profiles-dir "./ferryhopper"                                                                                                                                                                                main [3a3a10e9] modified
    ________                          __                  
   / ____/ /__  ____ ___  ___  ____  / /_____ ________  __
  / __/ / / _ \/ __ `__ \/ _ \/ __ \/ __/ __ `/ ___/ / / /
 / /___/ /  __/ / / / / /  __/ / / / /_/ /_/ / /  / /_/ / 
/_____/_/\___/_/ /_/ /_/\___/_/ /_/\__/\__,_/_/   \__, /  
                                                 /____/   

You are using Elementary 0.16.0, however version 0.16.1 is available.
Consider upgrading by running: "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/bin/python3.10 -m pip install --upgrade elementary-data"

Any feedback and suggestions are welcomed! join our community here - https://bit.ly/slack-elementary

2024-09-25 12:06:18 — INFO — Running with edr=0.16.0
2024-09-25 12:06:18 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path './ferryhopper' does not exist.
2024-09-25 12:06:18 — ERROR — Failed to parse Elementary's database and schema.
2024-09-25 12:06:18 — INFO — Running dbt command --log-format json run-operation elementary.log_macro_results --args {"macro_name": "elementary_cli.get_test_results", "macro_args": {"days_back": 7, "invocations_per_test": 720, "disable_passed_test_metrics": false}} --project-dir /home/tzanakis/git/python/pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/dbt_project --profiles-dir ./ferryhopper
2024-09-25 12:06:18 — ERROR — Could not generate the report - Error: Failed to run dbt command.
Path './ferryhopper' does not exist.
Please reach out to our community for help with this issue.
Traceback (most recent call last):
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/report/report.py", line 50, in get_report_data
    tests_api = TestsAPI(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 43, in __init__
    self.test_results_db_rows = self._get_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/api/tests/tests.py", line 55, in _get_test_results_db_rows
    return self.tests_fetcher.get_all_test_results_db_rows(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/monitor/fetchers/tests/tests.py", line 25, in get_all_test_results_db_rows
    run_operation_response = self.dbt_runner.run_operation(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 175, in run_operation
    result = self._run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 112, in _run_command
    result = self._inner_run_command(
  File "/home/tzanakis/.pyenv/versions/3.10.11/envs/dbt_test_upgrade_2-3.10.11/lib/python3.10/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path './ferryhopper' does not exist.

Let me know if you need to me to try anything else!

geo909 avatar Sep 25 '24 09:09 geo909

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them) @geo909

ofek1weiss avatar Sep 25 '24 13:09 ofek1weiss

ohhhhhhh, you use elementary 0.16.0, in 0.16.1 it should work (it is the only difference between them) @geo909

Ough.. I had tried with 0.16.1 earlier (see here) with the error again but I must have switched to 0.16.0 when playing around with the versions. Apologies again!

But we're getting there. Indeed now 0.16.1 works with the following combination:

  • edr report --project-dir "ferryhopper" --profiles-dir "ferryhopper"
  • keyfile: "/home/tzanakis/Tmp/bi.bak/dbt/ferryhopper/service-account-dbt.json"

But if I switch the keyfile parameter in profiles.yml to simply keyfile: service-account-dbt.json it is going to fail again.

Note that dbt runs fine with dbt run --project-dir "ferryhopper" --profiles-dir "ferryhopper" with the parameter set as keyfile: service-account-dbt.json.

geo909 avatar Sep 27 '24 05:09 geo909

@ofek1weiss I cloned elementary's repo and played around, the issue is with this commit.

If you comment-out the line cwd=self.project_dir, my edr command works fine with keyfile: service-account-dbt.json, put it back and it breaks again.

Edit: Tried with the latest version of elementary, and it's the same if I comment out this line, this is what brakes it for me.

geo909 avatar Sep 27 '24 13:09 geo909

@geo909 i see, thanks for the deep dive 🙏 this line was added to overcome this issue in dbt, a fix for it has been merged, but they did not yet release the fixed version, once they do we will be able to change it

ofek1weiss avatar Sep 29 '24 07:09 ofek1weiss

@geo909 i see, thanks for the deep dive 🙏 this line was added to overcome this issue in dbt, a fix for it has been merged, but they did not yet release the fixed version, once they do we will be able to change it

Great, good to know! Do you have an estimate about how long this could take, by any chance?

geo909 avatar Sep 29 '24 07:09 geo909

Same issue for me after upgrading to 0.16.1 Do you have any estimation yet?

FelixJetschiny avatar Dec 17 '24 15:12 FelixJetschiny

@ofek1weiss Any updates on this?

This is actually blocking us from upgrading to 0.19 and we're trying workarounds!

geo909 avatar Jun 26 '25 14:06 geo909