Support Min/Max Threshold for Custom SQL Tests #14983

[Anixton]

trafficstars

Describe your changes:

Fixes #14983

I worked on Support Min/Max Threshold for Custom SQL Tests because it's good first issue.

Type of change:

• [ ] Bug fix
• [ ] Improvement
• [ ] New feature
• [x] Breaking change (fix or feature that would cause existing functionality to not work as expected)
• [ ] Documentation

Checklist:

• [x] I have read the CONTRIBUTING document.
• [ ] My PR title is Fixes <issue-number>: <short explanation>
• [ ] I have commented on my code, particularly in hard-to-understand areas.
• [ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

[github-actions[bot]]

Hi there 👋 Thanks for your contribution!

The OpenMetadata team will review the PR shortly! Once it has been labeled as safe to test, the CI workflows will start executing and we'll be able to make sure everything is working as expected.

Let us know if you need any help!

[pmbrull]

Thanks for the PR @Anixton! Let us take a look here and bring this in! Appreciate it! cc @TeddyCr

[github-actions[bot]]

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[TeddyCr]

@Anixton thanks for the PR. The changes you made are not in the correct file. You will need to modify things in:

• https://github.com/open-metadata/OpenMetadata/blob/37a7cf99b66fd39aaedbf520d6c5d82837936477/openmetadata-service/src/main/resources/json/data/tests/tableCustomSQLQuery.json
• https://github.com/open-metadata/OpenMetadata/blob/d0cffdcd66542834662e9714117cd5ec03294aeb/ingestion/src/metadata/data_quality/validations/table/base/tableCustomSQLQuery.py

Basically replacing the threshold with minThreshold and maxThreshold and implementing the logic so that we can enforce the behavior either when both are set or just one is set.

[github-actions[bot]]

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions[bot]]

The Java checkstyle failed.

Please run mvn spotless:apply in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Java code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[Anixton]

@TeddyCr Hey I did some changes. Can you check if my modifications are correct?

[TeddyCr]

@TeddyCr Hey I did some changes. Can you check if my modifications are correct?

thanks for the update. It is partially correct. You also need to add the logic on the python side of things to handle the result processing:

• https://github.com/open-metadata/OpenMetadata/blob/d0cffdcd66542834662e9714117cd5ec03294aeb/ingestion/src/metadata/data_quality/validations/table/base/tableCustomSQLQuery.py

[Anixton]

@TeddyCr Hey I did some changes. Can you check if my modifications are correct?

thanks for the update. It is partially correct. You also need to add the logic on the python side of things to handle the result processing:

• https://github.com/open-metadata/OpenMetadata/blob/d0cffdcd66542834662e9714117cd5ec03294aeb/ingestion/src/metadata/data_quality/validations/table/base/tableCustomSQLQuery.py

@TeddyCr I added the logic on python side.

[sonarqubecloud[bot]]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
No data about Coverage
No data about Duplication

See analysis details on SonarCloud

[cypress[bot]]

16 failed tests on run #33303 ↗︎

16

715

12

0

0

Details:

Merge branch 'main' into patch-1
Project: openmetadata	Commit: 9dc914327d
Status: Failed	Duration: 10:20 💡
Started: Mar 3, 2024 5:55 PM	Ended: Mar 3, 2024 7:05 PM

  Service/ServiceIngestion.spec.ts • 3 failed tests • cypress-ci-mysql-pr-8131933068-1

View Output Video

Test		Artifacts
Superset Ingestion > Create & Ingest Superset service		Screenshots Video
Superset Ingestion > Update schedule options and verify		Screenshots Video
Superset Ingestion > Delete Superset service		Screenshots Video

  Pages/DataQualityAndProfiler.spec.js • 2 failed tests • cypress-ci-mysql-pr-8131933068-1

View Output Video

Test		Artifacts
Data Quality and Profiler should work properly > SQL query should be visible while editing the test case		Screenshots Video
Data Quality and Profiler should work properly > Update profiler setting modal		Screenshots Video

  Pages/Teams.spec.js • 1 failed test • cypress-ci-mysql-pr-8131933068-1

View Output Video

Test		Artifacts
Teams flow should work properly > Update description for created team		Screenshots Video

  Pages/Tags.spec.js • 1 failed test • cypress-ci-mysql-pr-8131933068-1

View Output Video

Test		Artifacts
Classification Page > Should render basic elements on page		Screenshots Video

  Features/QueryEntity.spec.js • 1 failed test • cypress-ci-mysql-pr-8131933068-1

View Output Video

Test		Artifacts
Query Entity > Update owner, description and tag		Screenshots Video

The first 5 failed specs are shown, see all 11 specs in Cypress Cloud.

Review all test suite changes for PR #15136 ↗︎