data-diff icon indicating copy to clipboard operation
data-diff copied to clipboard

Resolve failing mssql tests

Open pik94 opened this issue 2 years ago • 1 comments

This PR affects MSSQL connector

  • Fix timestamp normalization
  • Fix number normalization
  • Partly fix uuid normalization: text UUID are supported, native UUID is turned of

The reason why i decided to exclude native UUID type from the current implementation is casting. MSSQL UUID values are stored in upper case, whereas other databases generally use lower case. Moreover, when casting MSSQL UUIDs to python UUIDs, we lose upper case, and it leads to incorrect sql query gereration. For example, we want to compare MSSQL vs PostgreSQL tables with UUID primary keys. For MSSQL a query which calculates md5 hash looks like:

SELECT 
    COUNT(*), 
    MD5(... [col] ...) 
FROM UUID_TABLE 
WHERE ([col] >= 'af8ef74d-9b83-11ee-8e2a-000000000000') AND ([col] < 'af8efd8d-f0d8-6743-ef3c-000000000011')

UUID values in WHERE clause are in lower case after query building, whereas we need upper case for them, and as the result, we have an empty sample in contrast to PostgreSQL.

I have not managed to fix this on the MSSQL connection sidbe, it should be done in the core side.

pik94 avatar Dec 19 '23 15:12 pik94

Related:

  • Varchar(max) https://github.com/datafold/data-diff/pull/825
  • UUID case preservation https://github.com/datafold/data-diff/pull/825

nolar avatar Jan 02 '24 12:01 nolar

This pull request has been marked as stale because it has been open for 60 days with no activity. If you would like the pull request to remain open, please comment on the pull request and it will be added to the triage queue. Otherwise, it will be closed in 7 days.

github-actions[bot] avatar Mar 03 '24 06:03 github-actions[bot]

Although we are closing this pull request as stale, it's not gone forever. PRs can be reopened if there is renewed community interest. Just add a comment and it will be reopened for triage.

github-actions[bot] avatar Mar 11 '24 06:03 github-actions[bot]