feat/430/handle custom schemas

[PaulMatthieu]

Description

DBT lets you use custom schemas. This add the ability for dbt-sugar to parse those custom schemas and find the correct tables & columns during tasks.

How was the change tested

I tested it against our analytics db at Withings but their are multiple ways to use custom schemas so I probably should add some functional tests based on maintainers feedbacks.

Issue Information

Closes #430

Checklist

• [x] I formatted my PR name according to CONTRIBUTING.md
• [ ] I added a news fragment to help populating the changelog as encouraged in CONTRIBUTING.md
• [x] I added "Closes #<issue_number>" in the "Issue Information" section (if no issue, feel free to tick thick the box anyway).

[sourcery-ai[bot]]

Sourcery Code Quality Report

❌  Merging this PR will decrease code quality in the affected files by 0.06%.

Quality metrics	Before	After	Change
Complexity	6.47 ⭐	6.85 ⭐	0.38 👎
Method Length	64.11 🙂	65.88 🙂	1.77 👎
Working memory	8.22 🙂	7.85 🙂	-0.37 👍
Quality	69.82% 🙂	69.76% 🙂	-0.06% 👎

Other metrics	Before	After	Change
Lines	1405	1490	85

Changed files	Quality Before	Quality After	Quality Change
dbt_sugar/core/clients/yaml_helpers.py	70.47% 🙂	67.56% 🙂	-2.91% 👎
dbt_sugar/core/task/audit.py	76.46% ⭐	76.46% ⭐	0.00%
dbt_sugar/core/task/base.py	69.58% 🙂	70.26% 🙂	0.68% 👍
dbt_sugar/core/task/bootstrap.py	70.07% 🙂	69.96% 🙂	-0.11% 👎
dbt_sugar/core/task/doc.py	64.80% 🙂	64.53% 🙂	-0.27% 👎
tests/base_task_test.py	77.70% ⭐	78.48% ⭐	0.78% 👍

Here are some functions in these files that still need a tune-up:

File	Function	Complexity	Length	Working Memory	Quality	Recommendation
dbt_sugar/core/task/base.py	BaseTask.find_model_schema_file	25 😞	140 😞	13 😞	37.64% 😞	Refactor to reduce nesting. Try splitting into smaller methods. Extract out complex expressions
dbt_sugar/core/task/base.py	BaseTask.update_model_description_test_tags	25 😞	128 😞	11 😞	42.34% 😞	Refactor to reduce nesting. Try splitting into smaller methods. Extract out complex expressions
dbt_sugar/core/task/doc.py	DocumentationTask.check_tests	12 🙂	173 😞	13 😞	43.95% 😞	Try splitting into smaller methods. Extract out complex expressions
dbt_sugar/core/task/doc.py	DocumentationTask.update_model	14 🙂	97 🙂	11 😞	54.29% 🙂	Extract out complex expressions
dbt_sugar/core/task/bootstrap.py	BootstrapTask.add_or_update_model_descriptor_placeholders	12 🙂	115 🙂	10 😞	55.51% 🙂	Extract out complex expressions

Legend and Explanation

The emojis denote the absolute quality of the code:

• ⭐ excellent
• 🙂 good
• 😞 poor
• ⛔ very poor

The 👍 and 👎 indicate whether the quality has improved or gotten worse with this pull request.

---

Please see our documentation here for details on how these metrics are calculated.

We are actively working on this report - lots more documentation and extra metrics to come!

Help us improve this quality report!

[bastienboutonnet]

Thanks for opening this PR @PaulMatthieu I'll have a look in the coming days and see what we can do to get this merged as soon as possible.

[PaulMatthieu]

I realized while testing this that this change ignore whatever is written in the {{ config() }} bloc in a dbt model
So for instance if your target schema is foo and you write {{ config(schema='bar') }} dbt-sugar will still look into the foo schema instead of the foo_bar schema, custom schemas will have to be defined in the dbt_project.yml file

[bastienboutonnet]

Thanks a lot for this @PaulMatthieu

I think this is totally a welcome change!

I realized while testing this that this change ignore whatever is written in the {{ config() }} bloc in a dbt model So for instance if your target schema is foo and you write {{ config(schema='bar') }} dbt-sugar will still look into the foo schema instead of the foo_bar schema, custom schemas will have to be defined in the dbt_project.yml file

That's indeed a problem because we re-implemented everything back in the day when the dbt API wasn't really stable.

I think the best way to really tackle this would be to leverage the dbt API more now that it's stable. But that's quite a bigger thing.

My guess, here, would be that we have to temporarily document that limitation. Do I get that right?

[bastienboutonnet]

Btw, I'm not sure if you say but a bunch of unit tests are failing on this PR, do you need help with it?