dbt-loom icon indicating copy to clipboard operation
dbt-loom copied to clipboard

[Feature] Create new configuration abstraction for a Project

Open nicholasyager opened this issue 9 months ago • 5 comments

Is your feature request related to a problem? Please describe. Currently, dbt-loom does not have a good way to leverage a dbt project's restrict-access configuration. This config is important for limiting cross-project access to protected and private nodes. Since this is an entirely new file to load, we do not currently have a defined path for loading this information.


flowchart LR
  
  subgraph project_a
    dbt_project.yml
    manifest.json
  end

  subgraph project_b
    project_b_project[dbt_project.yml]
    project_b_manifest[manifest.json]
    dbt_loom.config.yml

    subgraph dbt-loom

      ManifestLoader
    
      Plugin 

    end
  end

  
  manifest.json --> ManifestLoader

  
  ManifestLoader --> Plugin

  dbt_loom.config.yml --> Plugin


  dbt-core

  Plugin --> dbt-core
  project_b_project --> dbt-core
  project_b_manifest --> dbt-core

Describe the solution you'd like I'd like to be able to define a Project, and with this project a location for its dbt_project.yml file and an associated manifest.json file. This should support all of our existing artifact sources where possible.


flowchart LR
  
  subgraph project_a
    dbt_project.yml
    manifest.json
  end

  subgraph project_b
    project_b_project[dbt_project.yml]
    project_b_manifest[manifest.json]
    dbt_loom.config.yml

    subgraph dbt-loom

      ManifestLoader
      ProjectLoader
      Plugin 

    end
  end

  dbt_project.yml --> ProjectLoader
  manifest.json --> ManifestLoader

  ProjectLoader --> Plugin
  ManifestLoader --> Plugin

  dbt_loom.config.yml --> Plugin


  dbt-core

  Plugin --> dbt-core
  project_b_project --> dbt-core
  project_b_manifest --> dbt-core

To configure this, we can introduce a new optional top-level concept of a Project.

dependencies:

  - name: core
    description: All common core dependencies across our `n` base projects.
    artifacts:
      - type: s3
        config:
          bucket_name: com.example.dbt_artifacts 
          object_prefix: latest/
        
   - name: revenue
     description: A proof-of-concept local-only revenue reporting project.
     artifacts:
        - type: file
          config:
            path: path/to/manifest.json
        - type: file
          config:
            path: path/to/dbt_project.yml       
   

Describe alternatives you've considered

  • Have restrict-access configured in the dbt_loom.config.yml file instead.
  • Don't support restruct-access at all.

Additional context

  • It would be ideal to keep the door open for bulk loading of artifacts, as described in #31

nicholasyager avatar May 13 '24 02:05 nicholasyager