windmill icon indicating copy to clipboard operation
windmill copied to clipboard

feature: Add a data catalog like openmetadata or from microsoft

Open suse-coder opened this issue 6 months ago • 3 comments

Add a data catalog, where users can upload data and when a user wants to use it it has to get approved. Would be a good fit as that is really what i miss in windmill. A little bit like openmetadata or in databricks catalog or Microsoft Purview/Onelake.

Would be great if one can have in Windmill one unified access point, where the different data connectors get the data, where in windmill it is unified in one data lake and one has one api and connection keys for each user (so one does not need to have it for every database hardcoded).

suse-coder avatar Jun 12 '25 18:06 suse-coder

Quilt data: https://docs.quilt.bio/quilt-platform-administrator/admin for example has a very innovative approach where everything is a package in s3 (versioning in s3 buckets), and the data catalog is then referring to that data packages.

suse-coder avatar Jun 13 '25 08:06 suse-coder

Like having a connector to oci images and have rbac to define (over sso or via groups in windmill) what one can publish and read of oci image (like juzu from kitops is doing it) https://jozu.com/product/ https://jozu.ml/browse

Image

suse-coder avatar Jun 14 '25 09:06 suse-coder

And please make it that windmill is federated, so one can connect to others and then in the serach result also get their data ressources (oci or others workflows)

suse-coder avatar Jun 14 '25 23:06 suse-coder