dbt-loom
dbt-loom copied to clipboard
[Feature] Create new configuration abstraction for a Project
Is your feature request related to a problem? Please describe.
Currently, dbt-loom does not have a good way to leverage a dbt project's restrict-access
configuration. This config is important for limiting cross-project access to protected
and private
nodes. Since this is an entirely new file to load, we do not currently have a defined path for loading this information.
flowchart LR
subgraph project_a
dbt_project.yml
manifest.json
end
subgraph project_b
project_b_project[dbt_project.yml]
project_b_manifest[manifest.json]
dbt_loom.config.yml
subgraph dbt-loom
ManifestLoader
Plugin
end
end
manifest.json --> ManifestLoader
ManifestLoader --> Plugin
dbt_loom.config.yml --> Plugin
dbt-core
Plugin --> dbt-core
project_b_project --> dbt-core
project_b_manifest --> dbt-core
Describe the solution you'd like
I'd like to be able to define a Project
, and with this project a location for its dbt_project.yml
file and an associated manifest.json
file. This should support all of our existing artifact sources where possible.
flowchart LR
subgraph project_a
dbt_project.yml
manifest.json
end
subgraph project_b
project_b_project[dbt_project.yml]
project_b_manifest[manifest.json]
dbt_loom.config.yml
subgraph dbt-loom
ManifestLoader
ProjectLoader
Plugin
end
end
dbt_project.yml --> ProjectLoader
manifest.json --> ManifestLoader
ProjectLoader --> Plugin
ManifestLoader --> Plugin
dbt_loom.config.yml --> Plugin
dbt-core
Plugin --> dbt-core
project_b_project --> dbt-core
project_b_manifest --> dbt-core
To configure this, we can introduce a new optional top-level concept of a Project.
dependencies:
- name: core
description: All common core dependencies across our `n` base projects.
artifacts:
- type: s3
config:
bucket_name: com.example.dbt_artifacts
object_prefix: latest/
- name: revenue
description: A proof-of-concept local-only revenue reporting project.
artifacts:
- type: file
config:
path: path/to/manifest.json
- type: file
config:
path: path/to/dbt_project.yml
Describe alternatives you've considered
- Have
restrict-access
configured in thedbt_loom.config.yml
file instead. - Don't support
restruct-access
at all.
Additional context
- It would be ideal to keep the door open for bulk loading of artifacts, as described in #31