dbt-external-tables icon indicating copy to clipboard operation
dbt-external-tables copied to clipboard

Feature/add ignore case option to snowflake

Open robby-rob-slalom opened this issue 10 months ago • 4 comments

Description & motivation

PR for Add ignore_case option to snowflake infer schema #288

Checklist

TODO:

  • [x] I have verified that these changes work locally
  • [ ] I have updated the README.md (if applicable)
  • [ ] I have added an integration test for my fix/feature (if applicable)

robby-rob-slalom avatar Apr 11 '24 23:04 robby-rob-slalom

the change seems straight forward enough. however we'll need at least one test case to prove this out. perhaps even a new seed table and external parquet file to test this against?

I'm not sure where this file would go but this is what I used to generate the example mixed case parquet (edit: modified to match existing schema in public_data):

-- partition with UPPERCASE column format
COPY INTO @stage/parquet_with_inferred_schema_and_mixed_column_case
FROM (
    SELECT
        1 AS "ID",
        'FOO' AS "NAME",
        'a' AS "SECTION"
)
PARTITION BY ('section="SECTION"')
FILE_FORMAT = (TYPE = PARQUET)
HEADER = TRUE
;

-- partition with lowercase column format
COPY INTO @stage/parquet_with_inferred_schema_and_mixed_column_case
FROM (
    SELECT
        2 AS "id",
        'bar' AS "name",
        'b' AS "section"
)
PARTITION BY ('section="section"')
FILE_FORMAT = (TYPE = PARQUET)
HEADER = TRUE
;

-- partition with PascalCase column format
COPY INTO @stage/parquet_with_inferred_schema_and_mixed_column_case
FROM (
    SELECT
        3 AS "Id",
        'FooBar' AS "Name",
        'c' AS "Section"
)
PARTITION BY ('section="Section"')
FILE_FORMAT = (TYPE = PARQUET)
HEADER = TRUE
;

robby-rob-slalom avatar Apr 12 '24 22:04 robby-rob-slalom

@robby-rob-slalom is this still a draft? or do you think it's ready to be "formally" reviewed?

dataders avatar May 01 '24 14:05 dataders

@robby-rob-slalom is this still a draft? or do you think it's ready to be "formally" reviewed?

The code change is ready. I may need some assistance with the integration test. Looking at the /public_data folder, there could be another folder like /json_mixed_case where one of the section files has uppercase keys and another has title case keys.

robby-rob-slalom avatar May 02 '24 15:05 robby-rob-slalom

Didn't realize a PR for ignore_case started, i have created this PR that handles this for both infer schema or when the schema is specified in the external table definition - https://github.com/dbt-labs/dbt-external-tables/pull/308 tagging you - @dataders

cakkinep avatar Jul 25 '24 11:07 cakkinep

Feature included in #308

robby-rob-slalom avatar Aug 20 '24 15:08 robby-rob-slalom