kedro-viz
kedro-viz copied to clipboard
Add Dataset Factory Patterns to Experiment Tracking
Experiment Tracking relies on Tracking Datasets and we create TrackingDatasetModel before we discover and get all the datasets that are defined using dataset factory pattern
Ideal way would be to implement this discovery on the kedro side, but if not possible, we need to discuss on how to discover the datasets. Earlier we used .exists() on the datasets for discovery which caused the below issues -
- Users were not able to use Kedro Viz if the dataset defined in the catalog does not exist (This was resolved using exception handlers)
- Users received a timed-out issue as the datasets resided remotely and in a distributed cluster
Parent issue - https://github.com/kedro-org/kedro-viz/issues/1480