kedro
kedro copied to clipboard
[DataCatalog]: Interface to request specific properties from catalog and datasets
Description
Users seek increased efficiency in accessing data through the public API. They suggest implementing a kind of GraphQL interface that allows them to request specific properties from catalog and datasets, reducing unnecessary data transmission.
We propose to explore the feasibility of implementation and integration of such an interface into the public API.
Partially related to https://github.com/kedro-org/kedro/issues/143
Context
- "If we want some information from catalog we would like to have kind of GraphQL interface where you ask for a few information, like let's say, passing as an argument and then it checks if that key is available and catalog itself resolves that and returns the details. So that it might be like a single interface or single API for interacting with catalog instead of customizing on top of it."
- "If there is some interface where we ask for certain properties which we need from the catalog and catalog provides us those details instead of creating a session context and then getting a catalog."