Hierarchical / nested asset groups

[sryza]

What we've heard

• https://elementl-workspace.slack.com/archives/C0453JAC8KD/p1685461491497549
  • Is there a way to create subgroups of assets? For example I have a wine quality classifier use case and I want to have a wine_qualitygroup, but within this I would like the training and prediction to be separate so I can use the Materialise all.

[mkleinbort-ic]

Hi there... checking in if this will be implemented within the next 3-12 months.

The ask is to have "nested" groups - namely

Last time this was discused we were talking about extending the group_name parameter of @asset to support lists

for example (in reference to the diagram above)

@asset(asset_goup=['rankingAPI'])
def select_features():
    ...

@asset(asset_goup=['rankingAPI'])
def serving_data(selected_features):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'DataPreparation'])
def get_feature1(selected_features):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'DataPreparation'])
def get_feature2(selected_features):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'DataPreparation'])
def training_data(get_feature1, get_feature2):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'MachineLearning'])
def X_train(training_data):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'MachineLearning'])
def y_train(training_data):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'MachineLearning'])
def x_test(training_data):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'MachineLearning'])
def y_test(training_data):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'MachineLearning'])
def model(X_train, y_train):
    ...

@asset(asset_goup=['rankingAPI', 'Modeling', 'MachineLearning'])
def_report(model, X_test, y_test):
    ...

@asset(asset_goup=['rankingAPI'])
def update_API(serving_data, model):
    ...

[indy-jonesy]

Will this support multiple sets of groups like the ML team and the Data Science team as both being tagged as "owners" of the Modeling nested groups? They could then filter to nested assets, that are associated to them.

I attest that this request would be hugely valuable once fulfilled.

[robolyst]

Is this something that is on the roadmap? As projects become larger, organisation really matters.

[rmmckinney96]

Hey guys we'd love an update on this. For our purposes we like using AssetKeys with lists to group things.

something like: CodeLocation / Group / SubGroup / ... / AssetName

Right now we'll use this in the AssetDefinition group_name: Group tags: SubGroup, ... name: AssetName key_prefix: CodeLocation / Group / SubGroup / ... /

Filtering with code location, tags, & group names works well to search, but visually it would be nice to have the collapsible hierarchy in the dagit UI. It would also be useful to have hierarchical groups rather than having to add tags, which removes the hierarchical relationship.

[braunjj]

Hey all, I really love this idea but it is not something the team currently has on the roadmap for the next release