elementary
elementary copied to clipboard
[Feature] End to end lineage graph - include BI and ETL tools
In order to make the graph more informative and useful, we want to present the BI and ETL tools on the lineage graph. The relevant tools are upstream (data sources) and downstream (consumers/destinations) from the DWH.
As we build lineage from the DWH available logs, and we don't want additional integrations at the moment, the idea is: The user will create a configuration file (YAML) that will associate service users to tools
For sources (Fivetran, AirByte, etc.) - The service users the tools use to load data to the DWH. For targets ( Looker, Tableau, etc.) - The service users the tools use to pull data from the DWH.
Example YAML format:
sources:
airbyte:
user_name: airbyte
role: airbyte_service
destinations:
tableau:
user_name:tableau
role:tableau_service
Feature output demo: