OpenMetadata
OpenMetadata copied to clipboard
open-metadata
table filter pattern: exclude empty tables
Is your feature request related to a problem? Please describe. I am working on ingesting metadata from COTS solutions. Those solution have many pre-built tables for many scenarios. For example I have a retail system that covers POS, Inventory, Scheduling, Staff. However, we only use it for POS transactions.
- The solution has over 700 tables
- Unfortunately all tables are under one schema.
- There is logic in table name pattern, but even within the POS related tables, we use 46% of tables
Why empty tables don't add value for data search and discovery?
- This increases the amount of "dead-end" search results. Person searches for table or column. Gets a result; based on on schema looks promising. Goes to see the data (maybe profile, maybe sample data) and finds that it is empty. First thought - sample data / profile wasn't ingested. Then sees it was ingested, and realizes that table is simply empty. they have to restart the search. I would find it very annoying.
- misrepresents "data asset" count.
Describe the solution you'd like On metadata ingestion, have an option to exclude tables with zero rows (mainly for BIG databases like oracle, mssql). So that I don't have to remove one by one.
Describe alternatives you've considered Run table metrics, identify all empty tables and have a very long "exclude by table name" filter pattern.
Additional context Add any other context or screenshots about the feature request here.
