soda-core icon indicating copy to clipboard operation
soda-core copied to clipboard

feat(profiling): add support for extra numeric and text datatypes

Open bastienboutonnet opened this issue 3 years ago • 3 comments
trafficstars

I recently realised that we were only considering the most common data types in column profiling for most dbs.

I've now added support to pretty much all numeric and text types for the following databases:

  • postgres (base) & therefore redshift
  • snowflake
  • bigquery
  • db2

I've left out:

  • athena because it seems like types for profiling were commented out (not sure why), see: https://github.com/sodadata/soda-core/blob/main/soda/athena/soda/data_sources/athena_data_source.py#L84-L85
  • sqlserver (all types seem to be already there from a previous implementation)
  • MySQL (all types seem to be already there from a previous implementation)

bastienboutonnet avatar Aug 11 '22 10:08 bastienboutonnet

There seems to be some failures for some data sources, although these seem to be due to issues with the docker containers. Not sure what I can do there other than re-triggering CI? @vijaykiran let me know if there's anything else I can do to help here.

bastienboutonnet avatar Aug 11 '22 10:08 bastienboutonnet

Looks like triggering the failed jobs again resolved the failing ones.

bastienboutonnet avatar Aug 11 '22 10:08 bastienboutonnet

Kudos, SonarCloud Quality Gate passed!    Quality Gate passed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 0 Code Smells

No Coverage information No Coverage information
0.0% 0.0% Duplication

sonarqubecloud[bot] avatar Aug 22 '22 06:08 sonarqubecloud[bot]