datahub
datahub copied to clipboard
feat(ingest): GE Profile/Action Trino support
Should resolve this too https://feature-requests.datahubproject.io/p/profiling-in-trino
Bumping GE version due to major fix on Trino connector here.
Feature update:
Trino will delete GE create temporary views
GE Actions with two new parameter refer to Documentation:
- exclude_dbname
- platform_alias
Test update to include those two new parameters
platform_alias
will be useful, when our main connector is presto-on-hive
and we are capitalizing on trino
to run test and profiling
Checklist
- [x] The PR conforms to DataHub's Contributing Guideline (particularly Commit Message Format)
- [x] Links to related issues (if applicable)
- [x] Tests for the changes have been added/updated (if applicable)
- [x] Docs related to the changes have been added/updated (if applicable). If a new feature has been added a Usage Guide has been added for the same.
- [x] For any breaking change/potential downtime/deprecation/big changes an entry has been made in Updating DataHub
Unit Test Results (build & test)
499 tests +409 499 :heavy_check_mark: +409 8m 15s :stopwatch: + 7m 52s 115 suites +101 0 :zzz: ± 0 115 files +101 0 :x: ± 0
Results for commit a6020c81. ± Comparison against base commit 355c129c.
:recycle: This comment has been updated with latest results.
Unit Test Results (metadata ingestion)
8 files ±0 8 suites ±0 1h 11m 4s :stopwatch: -49s 644 tests +1 642 :heavy_check_mark: +1 2 :zzz: ±0 0 :x: ±0 1 288 runs +2 1 284 :heavy_check_mark: +2 4 :zzz: ±0 0 :x: ±0
Results for commit a6020c81. ± Comparison against base commit 355c129c.
This pull request removes 1 and adds 2 tests. Note that renamed tests count towards both.
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest[test_checkpoint-ge_mcps_golden.json]
tests.integration.great-expectations.test_great_expectations ‑ test_ge_ingest[test_checkpoint_2-ge_mcps_golden_2.json]
:recycle: This comment has been updated with latest results.
Most of the error are from this,
E from __future__ import annotations
E ^
E SyntaxError: future feature annotations is not defined
https://stackoverflow.com/a/61544901 which need at least python 3.7. I guess using GE will need at least 3.7 python version.
Having some CICD Integration errors too.
Hey @aezomz, we just merged in #5521 so we should be clear to finish up this PR
Thanks, let me fix those failure tonight.
Are we intending to merge this soon?
@aezomz just merged, sorry about the delay.