soda-core icon indicating copy to clipboard operation
soda-core copied to clipboard

Add ability to specify custom hostname for Snowflake connection

Open whummer opened this issue 1 year ago • 4 comments

Add ability to specify custom host/port for Snowflake connections.

Motivation: At LocalStack, we have recently started building a Snowflake emulator that allows running SF queries entirely on the local machine: https://blog.localstack.cloud/2024-05-22-introducing-localstack-for-snowflake/ . As part of building out a repo with sample applications (see here), we are putting together an illustrative Soda application that connects to the local Snowflake emulator (running on localhost) and performs various data quality checks.

We have been able to get it working already by manually patching the soda-core library in the local Pyhon path.

$ cat configuration.yml
 data_source local_snowflake:
   type: snowflake
   connection:
     host: snowflake.localhost.localstack.cloud
     port: 4566
     username: test
     password: test
     ...

... and then:

$ soda scan --verbose -d local_snowflake -c configuration.yml checks.yml
...
[16:25:11] Scan summary:
[16:25:11] 2/2 queries OK
[16:25:11]   local_snowflake.test_table.aggregation[0] [OK] 0:00:00.034070
[16:25:11]   local_snowflake.test_table.schema[test_table] [OK] 0:00:00.040392
[16:25:11] 1/3 checks PASSED:
[16:25:11]     test_table in local_snowflake
[16:25:11]       No NULL values [checks.yml] [PASSED]
[16:25:11]         check_value: 0
[16:25:11] 1/3 checks FAILED:
[16:25:11]     test_table in local_snowflake
[16:25:11]       Dataset is unreasonably small [checks.yml] [FAILED]
[16:25:11]         check_value: 2
[16:25:11] 1/3 checks NOT EVALUATED:
[16:25:11]     test_table in local_snowflake
[16:25:11]       No changes to schema [checks.yml] [NOT EVALUATED]
...

It would be awesome if this integration was provided by Soda-core out of the box! Looking forward to getting your feedback.

Please let me know where is the best place to add a description for these new attributes (happy to add a short note in the docs), or if this change should be covered by a unit test. Thanks 🙌

/cc @m1n0

whummer avatar Jun 26 '24 14:06 whummer