dbt-dry-run icon indicating copy to clipboard operation
dbt-dry-run copied to clipboard

fix: dbt-external-tables partitions

Open betosaulo-cw opened this issue 1 year ago • 1 comments

Description

This update makes changes to how we handle partitions in our project. Here's what's new:

  • We've added a partitions list in the ExternalConfig class. It's for keeping track of different sections of our data. By default, this list starts empty.
  • There's now a quick way to see partition details in ExternalConfig through partitions_map
  • We now consider partitions when we're organizing our data, making sure everything is in the right place.

Fixes #53

Checklist:

  • [x] I have run make verify and fixed any linting or test errors
  • [ ] I have added appropriate unit tests or if applicable an integration test
  • [ ] OPTIONAL: I have run make integration against a Big Query instance

betosaulo-cw avatar Dec 29 '23 13:12 betosaulo-cw

Looks like it will work! Can you add a test to check the dry run schema is as expected when partitions is used?

The test in here should be a good starting point but you will probably want to use assert_result_has_table like this to ensure the schema is correctly predicted:

# The test should pass if no InvalidColumnSpecification exception is raised
expected_table = Table(fields=[TableField(name="column1", type="STRING"), ..., ...])
assert_result_has_table(expected_table, result)

ccharlesgb avatar Jan 04 '24 11:01 ccharlesgb

Closing as this is stale. Would still accept a fresh PR 👍

ccharlesgb avatar May 14 '24 15:05 ccharlesgb