Add local file upload functionality to adapter.

[danthelion]

trafficstars

Partly resolves https://github.com/brooklyn-data/dbt_artifacts/issues/61

Description

Add local file upload functionality to the adapter. The main motivation for this is to make dbt_artifacts compatible with Databricks. In order to do so we need to get the generated artifacts first to dbfs, from where to dbt package will be able to pick them up as sources for the downstream models. Ideally this upload process should happen in the adapter so the whole process in environment-independent and will do the same on dbt Cloud, local runs, etc.

As I haven't found an official Python Databricks API client, I went with databricks-api, which generates the client based on the internals of the official databricks-cli tool.

To use via command line

dbt -d run-operation databricks__upload_file --args "{local_file_path: './sample_files/manifest.js'}"

dbt -d run-operation databricks__upload_file --args "{local_file_path: './sample_files/manifest.js', overwrite: True}"

or as a pre-hook

{{
    config(
        pre_hook=[
            "{{ databricks__upload_file(
                local_file_path='./sample_files/manifest.json'
            ) }}",
            "{{ databricks__upload_file(
                local_file_path='./sample_files/manifest.json',
                overwrite=True
            ) }}"
        ]
    )
}}

Checklist

• [*] I have run this code in development and it appears to resolve the stated issue
• [*] This PR includes tests, or tests are not required/relevant for this PR
• [ ] I have updated the CHANGELOG.md and added information about my change to the "dbt-databricks next" section.

[robertsaxby]

@superdupershant ptal

[danthelion]

Thanks for the comments @ueshin! Added my responses & new commits. The only thing I'm not exactly sure about is the implementation of an _execute_api wrapper, I am not yet that familiar with dbt adapter patterns (especially this one) but I would love to learn how to fit this into the existing codebase properly.

[github-actions[bot]]

This PR has been marked as Stale because it has been open for 180 days with no activity. If you would like the PR to remain open, please remove the stale label or comment on the PR.

[benc-db]

@danthelion is this still an outstanding ask, or has it been solved some other way in the meantime? Apologies for this PR getting lost in the shuffle as we've switched maintainers a few times in the last year.

[benc-db]

Closing as stale with no response from Dan.