elementary icon indicating copy to clipboard operation
elementary copied to clipboard

[Feature] End to end lineage graph - include BI and ETL tools

Open Maayan-s opened this issue 4 years ago • 0 comments

In order to make the graph more informative and useful, we want to present the BI and ETL tools on the lineage graph. The relevant tools are upstream (data sources) and downstream (consumers/destinations) from the DWH.

As we build lineage from the DWH available logs, and we don't want additional integrations at the moment, the idea is: The user will create a configuration file (YAML) that will associate service users to tools

For sources (Fivetran, AirByte, etc.) - The service users the tools use to load data to the DWH. For targets ( Looker, Tableau, etc.) - The service users the tools use to pull data from the DWH.

Example YAML format:

sources:
    airbyte:
    	user_name: airbyte
    	role: airbyte_service

destinations:
    tableau:
    	user_name:tableau
    	role:tableau_service

Feature output demo:

Maayan-s avatar Oct 19 '21 16:10 Maayan-s