dbt-loom
dbt-loom copied to clipboard
Feature: Support cross-project Group evaluation
Description and Motivation
Within dbt-core, it is possible to assign models and other resources to groups
. These groups can be used to restrict other downstream resources from ref
-ing the resource if they are not part of the same group. Due to the way that dbt-core instantiates ModelNodes
based on ModelNodeArgs
, it has not been possible to enforce group reference constraints across project boundaries, and ref
-ing protected nodes from across projects has historically yielded error messages lacking the group name of the upstream node (#59).
This pull request unblocks cross-project group evaluation by extending our internal LoomModelNodeArgs
construct to include a group
string, and wraps the existing ModelNode.from_args()
classmethod with an internal model_node_wrapper
, which inject the upstream node's group
string into the resulting ModelNode
. I also added a similar wrapper to ensure that downstream projects have their upstream-projects injected as valid group names during group validity checks. The metadata will NOT be passed into the downstream manifest, but this is a good first step.
Along the way, I found another defect in dbt-core's evaluation of access evaluation for private models, which prevents grouped private models in projects with restricted access from using generic tests (https://github.com/dbt-labs/dbt-core/issues/10134). This means that we cannot enforce checks against protected
models as described in #43. I'll restore this functionality once the underlying defect has been resolved in dbt-core.