profiles.yml in project directory not found on Linux (Ubuntu)

[kokorin]

Describe the bug EDR 0.16.1 doesn't find profiles.yml in current directory and complains about missing .dbt folder in user's home directory. The bug appears on Ubuntu and not on Windows.

Documentation says:

-p, --profiles-dir PATH         Which directory to look in for the
                                profiles.yml file. If not set, edr will look
                                in the current working directory first, then
                                HOME/.dbt/

EDR 0.15.2 works correct.

To Reproduce Steps to reproduce the behavior:

0. Use Ubuntu
1. Put profiles.yml in DBT project, not in ~/.dbt
2. Run Elementary
3. Build project
4. Run edr monitor ... without specifying --profiles-dir option
5. Error appears: Path '/root/.dbt' does not exist.

Expected behavior EDR sends monitoring notifications (if any).

Screenshots N/A

Environment (please complete the following information):

• Elementary CLI (edr) version: 0.16.1
• Elementary dbt package version: 0.16.0
• dbt version you're using: 1.8.3
• Data warehouse: Snowflake
• Infrastructure details: Ubuntu, Gitlab CI Runner (if that matters)

Additional context I believe the problem is in new DBT API runner. DBT internal API has a bug and it changes CWD after running dbt deps. Elementary runs several DBT commands, so I'm 80% sure it's the case.

Elementary log:

2024-08-09 06:45:53 — INFO — Running with edr=0.16.1
2024-08-09 06:45:53 — INFO — Installing packages for edr internal dbt package...
2024-08-09 06:45:53 — INFO — Running dbt command --log-format json deps --project-dir /usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project --target dev
2024-08-09 06:45:55 — ERROR — Unable to get the latest invocation: Failed to run dbt command.
Path '/root/.dbt' does not exist.
2024-08-09 06:45:55 — ERROR — Failed to parse Elementary's database and schema.
2024-08-09 06:45:55 — INFO — Running internal dbt run to populate alerts
2024-08-09 06:45:55 — INFO — Running dbt command run -m elementary_cli.alerts.alerts_v2 --project-dir /usr/local/lib/python3.11/site-packages/elementary/monitor/dbt_project --target dev --vars {"days_back": 1}
Traceback (most recent call last):
  File "/usr/local/bin/edr", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/cli/cli.py", line 67, in invoke
    return super().invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1666, in invoke
    rv = super().invoke(ctx)
         ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/click/decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/monitor/cli.py", line 364, in monitor
    success = data_monitoring.run_alerts(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/monitor/data_monitoring/alerts/data_monitoring_alerts.py", line 294, in run_alerts
    popopulated_data_successfully = self._populate_data(
                                    ^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/monitor/data_monitoring/alerts/data_monitoring_alerts.py", line 78, in _populate_data
    success = self.internal_dbt_runner.run(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 225, in run
    result = self._run_command(
             ^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/clients/dbt/command_line_dbt_runner.py", line 114, in _run_command
    result = self._inner_run_command(
             ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/elementary/clients/dbt/api_dbt_runner.py", line 53, in _inner_run_command
    raise DbtCommandError(
elementary.exceptions.exceptions.DbtCommandError: Failed to run dbt command.
Path '/root/.dbt' does not exist.

Would you be willing to contribute a fix for this issue? Yes.

[ofek1weiss]

Hi @kokorin, does providing --profiles-dir . solve this issue for you? trying to understand the root cause

[kokorin]

Sorry for not mentioning it, yes, explicit --profiles-dir fixes the issue

[ofek1weiss]

Alright then, i am not sure that this is an incorrect behavior, but that the previous behavior was a nice side effect that was documented. I believe the course of action here is just to fix the documentation

[Maayan-s]

@ofek1weiss I see that @kokorin is right and there is a bug in dbt: https://github.com/dbt-labs/dbt-core/issues/8997

[lokofoko]

Guys, can you update your docs if that is the case and specifiying a profiles-dir is a requirement, because your help message says that by default it looks into pwd. It is hard to troubleshoot that way.