FIXES 15215: First draft implementation on extracting Snowflake metadata incrementally

[IceS2]

Describe your changes:

In order to reduce the amount of requests to Snowflake and improve the Metadata extraction I created an IncrementalConfig to switch between doing full extractions and incremental ones using the SNOWFLAKE.ACCOUNT_USAGE.TABLES table.

The idea is that we can get the timestamp for the last successful pipeline run and based on that get all the Tables that had a DDL change in Snowflake:

• If the Pipeline didn't finished successfuly in the last X days -> Full Extraction
• If the Pipeline finished successfully in the last X days -> Incremental Extraction by fetching the Schema table list from the ones that appear on the SNOWFLAKE.ACCOUNT_USAGE.TABLES table.

This initial implementation tries to make the least possible change while implementing this but it could be further improved by querying it just once and store the information for all schemas or even use it to fetch the schemas changes as well in further iterations.

Type of change:

• [x] Improvement

Checklist:

• [x] I have read the CONTRIBUTING document.
• [x] My PR title is Fixes <issue-number>: <short explanation>
• [x] I have commented on my code, particularly in hard-to-understand areas.
• [ ] For JSON Schema changes: I updated the migration scripts or explained why it is not needed.

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[cypress[bot]]

14 failed tests on run #33381 ↗︎

14

719

12

0

0

Details:

Improve code based on sonarcloud input
Project: openmetadata	Commit: a5f989a155
Status: Failed	Duration: 19:06 💡
Started: Mar 11, 2024 7:47 AM	Ended: Mar 11, 2024 9:06 AM

  Service/ServiceIngestion.spec.ts • 1 failed test • cypress-ci-mysql-pr-8229138829-1

View Output Video

Test		Artifacts
Superset Ingestion > Create & Ingest Superset service		Screenshots Video

  Pages/Glossary.spec.ts • 7 failed tests • cypress-ci-mysql-pr-8229138829-1

View Output Video

Test		Artifacts
Glossary page should work properly > Approval Workflow for Glossary Term		Screenshots Video
Glossary page should work properly > Update glossary term		Screenshots Video
Glossary page should work properly > Request Tags workflow for Glossary		Screenshots Video
Glossary page should work properly > Assets Tab should work properly		Screenshots Video
Glossary page should work properly > Remove Glossary term from entity should work properly		Screenshots Video
Glossary page should work properly > Delete glossary term should work properly		Screenshots Video
Glossary page should work properly > Delete glossary should work properly		Screenshots Video

  Pages/Customproperties.spec.ts • 1 failed test • cypress-ci-mysql-pr-8229138829-1

View Output Video

Test		Artifacts
Custom Properties should work properly > Custom properties for glossary and glossary terms > Add update and delete String, Integer, Markdown custom properties for glossary term		Screenshots Video

  Pages/Teams.spec.ts • 1 failed test • cypress-ci-mysql-pr-8229138829-1

View Output Video

Test		Artifacts
Teams flow should work properly > Update description for created team		Screenshots Video

  Flow/AddRoleAndAssignToUser.spec.ts • 1 failed test • cypress-ci-mysql-pr-8229138829-1

View Output Video

Test		Artifacts
Test Add role and assign it to the user > Create role		Screenshots Video

The first 5 failed specs are shown, see all 7 specs in Cypress Cloud.

Review all test suite changes for PR #15201 ↗︎

[github-actions[bot]]

The Python checkstyle failed.

Please run make py_format and py_format_check in the root of your repository and commit the changes to this PR. You can also use pre-commit to automate the Python code formatting.

You can install the pre-commit hooks with make install_test precommit_install.

[sonarqubecloud[bot]]

Quality Gate passed for 'open-metadata-ingestion'

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
55.8% Coverage on New Code
3.0% Duplication on New Code

See analysis details on SonarCloud