dbt-external-tables
dbt-external-tables copied to clipboard
Feature/add ignore case option to snowflake
Description & motivation
PR for Add ignore_case option to snowflake infer schema #288
Checklist
TODO:
- [x] I have verified that these changes work locally
- [ ] I have updated the README.md (if applicable)
- [ ] I have added an integration test for my fix/feature (if applicable)
the change seems straight forward enough. however we'll need at least one test case to prove this out. perhaps even a new seed table and external parquet file to test this against?
I'm not sure where this file would go but this is what I used to generate the example mixed case parquet (edit: modified to match existing schema in public_data):
-- partition with UPPERCASE column format
COPY INTO @stage/parquet_with_inferred_schema_and_mixed_column_case
FROM (
SELECT
1 AS "ID",
'FOO' AS "NAME",
'a' AS "SECTION"
)
PARTITION BY ('section="SECTION"')
FILE_FORMAT = (TYPE = PARQUET)
HEADER = TRUE
;
-- partition with lowercase column format
COPY INTO @stage/parquet_with_inferred_schema_and_mixed_column_case
FROM (
SELECT
2 AS "id",
'bar' AS "name",
'b' AS "section"
)
PARTITION BY ('section="section"')
FILE_FORMAT = (TYPE = PARQUET)
HEADER = TRUE
;
-- partition with PascalCase column format
COPY INTO @stage/parquet_with_inferred_schema_and_mixed_column_case
FROM (
SELECT
3 AS "Id",
'FooBar' AS "Name",
'c' AS "Section"
)
PARTITION BY ('section="Section"')
FILE_FORMAT = (TYPE = PARQUET)
HEADER = TRUE
;
@robby-rob-slalom is this still a draft? or do you think it's ready to be "formally" reviewed?
@robby-rob-slalom is this still a draft? or do you think it's ready to be "formally" reviewed?
The code change is ready. I may need some assistance with the integration test. Looking at the /public_data folder, there could be another folder like /json_mixed_case where one of the section files has uppercase keys and another has title case keys.
Didn't realize a PR for ignore_case started, i have created this PR that handles this for both infer schema or when the schema is specified in the external table definition - https://github.com/dbt-labs/dbt-external-tables/pull/308 tagging you - @dataders
Feature included in #308