Visualizations Tab in the Dashboard

[Cahllagerfeld]

[htahir1]

@znegrin where can this tab go in the dashboard? we wanted to put it in deployments but in principal we can do it everywhere right?

[znegrin]

@htahir1 this one was supposed to be deployment-specific, if I'm correct. The idea of dashboards is wider, and I would stick it to the project view (and even user-scoped). I thought we could go with something similar to what we showed for this visualizations tab, as the wider concept of dashboards is much larger.

[htahir1]

@znegrin I see. My conern is: the biggest limitation we have with our current version of "dashboard" is:

a) They are static -> I think most artifacts produce visualizations that dont change b) They have to be created BEFOREHAND and always within a pipeline run -> you cant just create them directly in the dashboard

These are quite significant limitations -> How do you think we can / should change it?

[znegrin]

@htahir1 I would rather create visualizations that are not attached to artifacts, and use different metrics, metadata or other information that is more useful and dynamic with presets and widgets (role-fit widgets beat raw artifact views for SMEs and PMs). But if we go with the artifact visualizations embedded, maybe what we can do is to give the option to always show the most recent version, and then the selected metrics associated with it, and the ability to track them over time (latest snapshot plus trends builds confidence without forcing users into raw traces).

I see the “artifact visualizations” approach as a first MVP to provide something to visualize. Focusing on the insights based on users from the evals course, dashboards should also include tracking of metrics related to cost, evals, RAG data health, or retriever and generator blocks, rather than only screenshots. But this needs a longer process of thinking and preparing. I added a Notion page with examples of possible widgets that GPT generated for me based on the insights from the evals course which can guide us as inspiration: https://www.notion.so/zenml/Custom-Dashboards-Visualizations-2a0f8dff253880d794c6d04400344ce8?source=copy_link.

[htahir1]

@znegrin yes i know what the longer term vision would be but im wondering what a minimal lovable version can be

[znegrin]

@htahir1 I thought showing the artifact visualizations was somehow a requirement.

I think some minimal widgets we can add include:

• A bar chart with the runs timeline (sorted by error or just showing the total runs).

• Allow users to add filtered tables (for example, the latest invocation runs created with specific filters, or a list of recent errors).

• Allow users to select metrics from the metadata and show them as simple KPIs or over time in a chart (selecting a timeframe). At first, we can use just one chart type (for example, line charts), and then in the future also give users the option to select other charts.

• Even if the visualizations are not dynamic, I guess it is still useful to allow showing the latest version of selected artifact visualizations. These widgets can then maybe have a dropdown to switch between versions or a button to compare versions.

And then I guess it would also be relevant to:

• Provide the option to set a name (+ description) for these dashboards.
• Allow multiple dashboards per project.
• Provide the option to set a name (+ description) for each widget.
• Provide a very simple date filter (maybe at first just with closed options like "Last 30 days", "Last 7 days",...
• The option to share them with other team members and add specific permissions to access them via RBAC.