Generate model "button" not working when using dbt-athena adapter (CatalogId and DatabaseName are None)

[wtfzambo]

Expected behavior

When clicking "Generate model" from source, the model is properly generated:

Actual behavior

Clicking generate model generates two error messages:

Traceback (most recent call last):
File "/home/vscode/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.20.0/dist/node_python_bridge.py", line 104, in <module>
value = eval(_compile(data['code'], '<input>', 'eval'), _locals) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<input>", line 1, in <module> File "/home/vscode/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.20.0/dist/dbt_integration.py", line 501, in get_columns_in_relation
return self.adapter.get_columns_in_relation(relation)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/workspace/.venv/lib/python3.11/site-packages/dbt/adapters/athena/impl.py", line 885, in get_columns_in_relation
table = glue_client.get_table(CatalogId=catalog_id, DatabaseName=relation.schema, Name=relation.identifier)[
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^...

and following

An error occured while trying to generate the model Parameter validation failed:
Invalid type for parameter CatalogId, value: None, type: <class 'NoneType'>, valid types: <class 'str'>
Invalid type for parameter DatabaseName, value: None, type: <class 'NoneType'>, valid types: <class 'str'>

Other buttons work fine (e.g. Run model, build model etc...)

Steps To Reproduce

The dbt project is inside a folder in the root of the repo. E.g.

my_big_project/
  other_stuff...
  dbt/
    models/
    dbt_project.yml
    ...

I'm using vscode devcontainer and the following configurations:

dockerfile

FROM python:3.11-slim-bookworm

ARG USERNAME=vscode
ARG USER_UID=1000
ARG USER_GID=${USER_UID}
ARG HOME=/home/${USERNAME}
ENV PATH=${HOME}/.local/bin:$PATH

RUN groupadd --gid $USER_GID $USERNAME \
    && useradd --uid $USER_UID --gid $USER_GID -m $USERNAME \
    && apt-get update -y \
    && apt-get upgrade -y \
    && apt-get install -y curl wget zsh git ca-certificates gnupg sudo \
    # [Optional] Add sudo support. Omit if you don't need to install software after connecting
    && echo $USERNAME ALL=\(root\) NOPASSWD:ALL > /etc/sudoers.d/${USERNAME} \
    && chmod 0440 /etc/sudoers.d/${USERNAME}

# Insall node and npm as it's needed for SQLTools
ARG NODE_MAJOR=18
RUN mkdir -p /etc/apt/keyrings \
    && curl -fsSL https://deb.nodesource.com/gpgkey/nodesource-repo.gpg.key | sudo gpg --dearmor -o /etc/apt/keyrings/nodesource.gpg \
    && echo "deb [signed-by=/etc/apt/keyrings/nodesource.gpg] https://deb.nodesource.com/node_$NODE_MAJOR.x nodistro main" | sudo tee /etc/apt/sources.list.d/nodesource.list \
    && apt-get install --no-install-recommends -y nodejs npm

RUN wget https://github.com/mikefarah/yq/releases/latest/download/yq_linux_amd64 -O /usr/bin/yq &&\
    chmod +x /usr/bin/yq

# Install poetry
ARG POETRY_VERSION="none"
RUN if [ "${POETRY_VERSION}" != "none" ]; then echo "Installing poetry" \
    && curl -sSL https://install.python-poetry.org | POETRY_VERSION=${POETRY_VERSION} python3 - ; \
    fi

USER ${USERNAME}

My profiles.yml

myproject:
  target: dev
  outputs:
    dev:
      database: awsdatacatalog
      aws_profile_name: myprofile
      region_name: eu-west-1
      s3_data_dir: s3://mybucket
      s3_staging_dir: s3://aws-athena-query-results-xxxxx
      schema: my_schema
      threads: 4
      type: athena

The rest is pretty much standard dbt project: I have a schema on AWS athena with some tables, and when I clicking "generate model" in the sources.yml file, it behaves as described.

Log output/Screenshots

This is as much as I have in term of logs, it seems like the "output" tab in vscode doesn't have the logs from this extension.

Operating System

Debian GNU/Linux 12 (bookworm)

dbt version

1.6.4

dbt Adapter

dbt-athena

dbt Power User version

0.20.1

Are you willing to submit PR?

• [X] Yes I am willing to submit a PR!

I would like to help and contribute if you can help me in pinpointing what could be causing this specific issue.

[lionsardesai]

Hi,

Thank you for filing this bug report. Please note that dbt-athena is a community-authored and community-maintained adapter for dbt. While it retains the signatures of functions defined in the dbt-core adapter, the implementation is quite different. Most adapters extend the macro-execution workflow defined in the base adapters.

It looks like athena is taking a different route altogether. Because of this, a few things have to be provided when generating models. From the sample snapshot of the sources file provided here, it seems in this case you need to supply both the database and the schema for dbt-athena functions to correctly identify the relation and to correctly generate the models. Modify the sources file to include those two parameters and try again!

There should be no code modifications required. Should this not fix the issue, please reopen this issue and provide an example for us to investigate.

[wtfzambo]

@lionsardesai I tried your suggestion but I still get an error, just a different one:

sources.yml

errors:

Traceback (most recent call last):
File "/home/vscode/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.22.0/dist/node_python_bridge.py", line 104, in <module>
value = eval(_compile(data['code'], '<input>', 'eval'), _locals)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "<input>", line 1, in <module> File "/home/vscode/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.22.0/dist/dbt_integration.py", line 97, in to_dict
return [to_dict(val) for val in obj]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/vscode/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.22.0/dist/dbt_integration.py", line 97, in <listcomp>
return [to_dict(val) for val in obj]
^^^^^^^^^^^^
File "/home/vscode/.vscode-server/extensions/innoverio.vscode-dbt-power-user-0.22.0/dist/dbt_integration.py", line 99, in to_dict
return to_dict(vars(obj))
^^^^^^^^^^^^^^^^^^
File "/...

[wtfzambo]

PS @lionsardesai I don't have the button available to reopen the issue 😅

[anandgupta42]

@wtfzambo The issue you are getting seems to be related to #514 . We can keep this ticket closed and use that ticket instead.

[Fredahintz]

I'm experiencing a similar issue. I'm running dbt-core version 1.6.8 with the dbt-athena-community extension v1.6.4 for AWS Athena. I've been able to install the dbt-power-user extension, but I'm unable to use the "Generate Model" plug-in. An error pops up saying "Could not generate model!If the issue persists, please seek help in our dbt Community Slack channel #tools-dbt-power-user or report it on GitHub". Dbt debug shows no errors and there are no messages in the "Problems" tab in VS Code. I tried adding the schema and database to my source definition but got the same error.