dbt-loom icon indicating copy to clipboard operation
dbt-loom copied to clipboard

Feature: Support cross-project Group evaluation

Open nicholasyager opened this issue 9 months ago • 1 comments

Description and Motivation

Within dbt-core, it is possible to assign models and other resources to groups. These groups can be used to restrict other downstream resources from ref-ing the resource if they are not part of the same group. Due to the way that dbt-core instantiates ModelNodes based on ModelNodeArgs, it has not been possible to enforce group reference constraints across project boundaries, and ref-ing protected nodes from across projects has historically yielded error messages lacking the group name of the upstream node (#59).

This pull request unblocks cross-project group evaluation by extending our internal LoomModelNodeArgs construct to include a group string, and wraps the existing ModelNode.from_args() classmethod with an internal model_node_wrapper, which inject the upstream node's group string into the resulting ModelNode. I also added a similar wrapper to ensure that downstream projects have their upstream-projects injected as valid group names during group validity checks. The metadata will NOT be passed into the downstream manifest, but this is a good first step.

Along the way, I found another defect in dbt-core's evaluation of access evaluation for private models, which prevents grouped private models in projects with restricted access from using generic tests (https://github.com/dbt-labs/dbt-core/issues/10134). This means that we cannot enforce checks against protected models as described in #43. I'll restore this functionality once the underlying defect has been resolved in dbt-core.

nicholasyager avatar May 13 '24 01:05 nicholasyager