Add tags and freshness to stg_sources

[alexfetterman]

Describe the feature

I'd like to be able to query data about source tags, and the freshness configuration of each source via the stg_sources model in the dbt Project Evaluator package.

Describe alternatives you've considered

Installing dbt-artifacts or elementary to be able to query source freshness configuration and source tags. We'd rather not introduce package bloat just to get at one field.

Additional context

Is this feature database-specific? Which database(s) is/are relevant? Please include any other relevant context here. We are using Snowflake.

Who will this benefit?

There are a number of different packages available that will auto-model metadata about the dbt project. We're trying to minimize overhead by installing a bunch of packages, and since the stg_sources model in the evaluator package is SO close to exactly what we need, it would be great if freshness and tags could be added to that model.

Are you interested in contributing this feature?

I would need a hand getting started on this if I were to contribute.

[dave-connors-3]

Hey @alexfetterman! definitely open to adding these fields! If you'd like to take a stab at a contribution, the steps look like

1. Adding the relevant fields here -- this is a model that generates 0 rows, and then gets values inserted in a post hook, so the goal is just to get the table schema determined.
2. Adding the logic to grab these fields from the source node in this macro -- these fields should be in the same order as the columns in the first step
3. Add these fields to int_all_graph_resources -- we generally try to bring everything into this table even if it's specific to a particular resource type
4. If you're feeling bold, you can add some freshness and some tags to the sources in the integration tests project and add a dbt test or two to prove we're extracting the data we expect!

let me know if a) you have any desire to tackle this, and if so b) if you need more guidance!

[alexfetterman]

Hi @dave-connors-3 thank you! This is my first contribution to an external project -- hope this is close.

I had trouble following the integration tests suggestion, and instead attempted to install my forked version of the project on a branch of our dbt repo (using this method, referencing the github url and hash of commit). When I tried to build the dbt_evaluator_package models referencing the feature branch of the forked project, the stg_sources model did not build successfully. I got a pretty strange error:

  001003 (42000): SQL compilation error:
  syntax error line 9,773 at position 75 unexpected '.'.
  syntax error line 9,799 at position 75 unexpected '.'.
  syntax error line 9,826 at position 75 unexpected '.'.
  syntax error line 9,852 at position 75 unexpected '.'.

There's certainly not 10k lines of code in that file so I'm a little confused.

Any ideas? Seems like a simple thing to change but I'm struggling on how to actually test that it's working.

[alexfetterman]

The draft PR is #447 linked above.