Fill in model/column descriptions when available in database metadata

[jenna-jordan]

Describe the feature

Model and column descriptions can be stored in the database using the persist_docs feature. It is sometimes useful to "regenerate" the YAML files for models when tweaks have been made. However, the regenerated YAML has empty descriptions, meaning you need to manually copy and paste the old descriptions. If these descriptions have been stored in the database (persist_docs is set to true), then that could be used as the source to fill in already written descriptions.

Describe alternatives you've considered

Manually copy and paste descriptions from the old yaml into the new yaml.

Additional context

Different databases may store these descriptions differently, but you should be able to reference how persist_docs handles it.

Who will this benefit?

Anybody using codegen to automate the creation of source/model YAML files with descriptions already written.

Are you interested in contributing this feature?

I'm not currently able to

[gwenwindflower]

Yep, makes sense, let me ask our Core DX embed how persist_docs works and see how feasible this is, but if it's relatively doable this totally makes sense, especially now that we're pulling descriptions from sources as well as of yesterday, it makes sense that sources+models should be able to pull from the db comments or metadata if it's not too gnarly to pull off.

[jgooly]

+1

In our case, we have several teams that consume Snowflake tables built by other teams (not using dbt) that have column descriptions as comments. We'd like to extract those comments as column descriptions when create the source yml files.

[github-actions[bot]]

This issue has been marked as Stale because it has been open for 180 days with no activity. If you would like the issue to remain open, please comment on the issue or else it will be closed in 7 days.

[jenna-jordan]

@dave-connors-3 any hope of this happening?

[Luiscri]

This would be so helpful for us when defining sources from external projects with a huge number of columns. Actually, I started using codegen thinking this was already implemented, as it would save my a considerable amount of time