kedro-viz icon indicating copy to clipboard operation
kedro-viz copied to clipboard

Add Dataset Factory Patterns to Experiment Tracking

Open ravi-kumar-pilla opened this issue 1 year ago • 0 comments

Experiment Tracking relies on Tracking Datasets and we create TrackingDatasetModel before we discover and get all the datasets that are defined using dataset factory pattern

Ideal way would be to implement this discovery on the kedro side, but if not possible, we need to discuss on how to discover the datasets. Earlier we used .exists() on the datasets for discovery which caused the below issues -

  1. Users were not able to use Kedro Viz if the dataset defined in the catalog does not exist (This was resolved using exception handlers)
  2. Users received a timed-out issue as the datasets resided remotely and in a distributed cluster

Parent issue - https://github.com/kedro-org/kedro-viz/issues/1480

ravi-kumar-pilla avatar Dec 18 '23 23:12 ravi-kumar-pilla