datahub icon indicating copy to clipboard operation
datahub copied to clipboard

feat(ingest): GE Profile/Action Trino support

Open aezomz opened this issue 2 years ago • 3 comments

Should resolve this too https://feature-requests.datahubproject.io/p/profiling-in-trino

Bumping GE version due to major fix on Trino connector here.

Feature update: Trino will delete GE create temporary views GE Actions with two new parameter refer to Documentation: - exclude_dbname - platform_alias

Test update to include those two new parameters

platform_alias will be useful, when our main connector is presto-on-hive and we are capitalizing on trino to run test and profiling

Checklist

  • [x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
  • [x] Links to related issues (if applicable)
  • [x] Tests for the changes have been added/updated (if applicable)
  • [x] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
  • [x] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub

aezomz avatar Jul 08 '22 06:07 aezomz

Unit Test Results (build & test)

499 tests  +409   499 :heavy_check_mark: +409   8m 15s :stopwatch: + 7m 52s 115 suites +101       0 :zzz: ±    0  115 files   +101       0 :x: ±    0 

Results for commit a6020c81. ± Comparison against base commit 355c129c.

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Jul 08 '22 17:07 github-actions[bot]

Unit Test Results (metadata ingestion)

       8 files  ±0         8 suites  ±0   1h 11m 4s :stopwatch: -49s    644 tests +1     642 :heavy_check_mark: +1  2 :zzz: ±0  0 :x: ±0  1 288 runs  +2  1 284 :heavy_check_mark: +2  4 :zzz: ±0  0 :x: ±0 

Results for commit a6020c81. ± Comparison against base commit 355c129c.

This pull request removes 1 and adds 2 tests. Note that renamed tests count towards both.
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest[test_checkpoint-ge_mcps_golden.json]
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest[test_checkpoint_2-ge_mcps_golden_2.json]

:recycle: This comment has been updated with latest results.

github-actions[bot] avatar Jul 08 '22 17:07 github-actions[bot]

Most of the error are from this,

E       from __future__ import annotations
E                                        ^
E   SyntaxError: future feature annotations is not defined

https://stackoverflow.com/a/61544901 which need at least python 3.7. I guess using GE will need at least 3.7 python version.

Having some CICD Integration errors too.

aezomz avatar Jul 12 '22 02:07 aezomz

Hey @aezomz, we just merged in #5521 so we should be clear to finish up this PR

hsheth2 avatar Aug 10 '22 22:08 hsheth2

Thanks, let me fix those failure tonight.

aezomz avatar Aug 11 '22 08:08 aezomz

Are we intending to merge this soon?

aezomz avatar Aug 16 '22 13:08 aezomz

@aezomz just merged, sorry about the delay.

hsheth2 avatar Aug 16 '22 16:08 hsheth2