Standalone static flowchart component for Kedro-viz

[rashidakanchwala]

Description

Kedro-Viz, or rather the flowchart of Kedro-Viz, is sometimes embedded in other websites. We have created a suboptimal developer experience doing that because developers who want to use Kedro-Viz this way have to interact with the Kedro-Viz server and specifically hide the parts of Kedro-Viz that they don't want to use.

We want to propose making the flowchart more modular because it is the most embedded feature and also figure out an easy way for users to generate HTML versions of the pipeline for sharing ease.

Additionally, there has been thought around creating %show_viz line magic to allow users to view a standalone static flowchart component in their Jupyter notebooks.

Possible Implementation

• We need to make a decision on what parts of the flowchart we want to embed given it is supposed to be static (e.g. side panel, metadata panel, show/hide layers, mini map, etc.)
• If %show_viz line magic is created we will additionally have to think about: "The static flowchart is not going to run on a server so how will we render it as an HTML in the Jupyter notebook?"

Checklist

• [x] Include labels so that we can categorise your feature request

[tynandebold]

Notes from a meeting on the issue

• Brix -> this is the current bad experience. They hide the bits of Viz they don't need.
• Platforms (sagemaker, DB, etc.) -> a version of Viz w/o a CLI, yet they still have full functionality. What's the experience? If we have %launch_viz, we might not need.
• %show_viz -> also useful in documentation.

Idea: Can we have a simple HTML file without any backend that can show the flowchart content without any functionality. Can only zoom in/out and drag around the flowchart. Can't change the data. Can't filter, etc.

Assumptions: users need this for documentation (can we just use the static image export??). Can Brix get away with using static HTML or do they need some of the toolbar functionality?

1. Platform user: %launch_viz solves it. This is covered here: https://github.com/kedro-org/kedro-viz/issues/910.
2. Brix use case: use as a React component. They're getting the whole app but only need parts. Q: do we need to keep the entire React component? Q: will Brix be happy with a HTML-generated view of the pipeline? This is a research task: #954 . Do you want both the whole Viz app or can we only give you a lite version? What functionality are you using now? Would you prefer to use a HTML component, why or why not?
3. Embed the flowchart in documentation in any arbitrary HTML file. For user in Jupyter notebooks, readme, documentation files. %show_viz and CLI commands to export a static image of the flowchart are the solves here (https://github.com/kedro-org/kedro-viz/issues/638). This is a GH issue.
4. Clean up Viz codebase, especially flowchart. What's the ideal final state for this? Tech design meeting is the first step in this process.

[rashidakanchwala]

So we did an investigation on how the flowchart component is used in Kedro-viz and how we can extract it so that any valid JSON (with nodes, edges, and layers) can connect to it.

Overview

Based on the investigation, we have made two conclusions:-

1. Extracting the flowchart components from Kedro-viz and creating a new github repo that only has the flowchart is relatively straightforward.

2. Integrating our current kedro-viz setup with this standalone flowchart component by removing redux is not recommended. This is mainly because it's a lot of work as redux is used heavily in Kedro-viz. The flowchart component in Kedro-viz has also reached a maturity and there's not much further work that can be done on it. It work's great and our users love it hence spending time and effort refactoring something that already works well and doesn't give any added benefit to our users is not a good idea.

If we still want to do the integration, we can do it in the future when we have the bandwidth. This wouldn't block the standalone component creation and we can easily do that first and do the integration much later.

This also means we might have 2 flowcharts to maintain but flowchart components have hardly been touched in the last two years; it is very low maintenance so we don't think having 2 repos would be a problem. However once we feel it is a big problem, then we can tackle the integration.

Details

Let's now look at the current flowchart setup in Kedro-viz and how the standalone setup will look in a bit more detail. Please refer to this miro(https://miro.com/app/board/uXjVPRvJR4w=/) board.

1. Current setup - In our current setup (first diagram on miro board), we first read the json from Kedro-viz backend, we then normalise all the json and store it in redux store. We then read the edges, nodes, and layer data from the redux store and use web workers to do some heavy computations required for the flowchart. This results in a graph object that is primarily used to create the d3 flowchart. In the miro-board, you can see the critical path for flowchart highlighted in yellow. The other states used are primarily for interactions with the sidebar and metadata panel (which we don't need in our standalone component)

2. Standalone flowchart setup - In this setup, we would read json from anywhere, validate it and normalise it, and put it in a flowchart object. We would then use web-workers to do the heavy computation and as a result receive a graph object. We would then create a d3 flowchart based on this graph object. In terms of implementation, we just need to copy the code from Kedro-viz and write some code to connect the different components.

3. Finally, the third diagram is just an idea of how we could possibly integrate the standalone with Kedro-viz. Maybe we could import the heavy computational code that is handled by web-workers from the standalone component to avoid having that code in two places.

Please let us know your thoughts - @idanov , @yetudada , @ottis , @NeroOkwa cc @tynandebold

[idanov]

Thanks for looking into this @rashidakanchwala, it's really an important and long-awaited work we've been delaying a lot. I have a couple of questions regarding your findings:

1. Could we create a separate package within the current repo instead of setting up a new repo?
2. Could we still make sure we use the same codebase for the flowchart without removing redux on Kedro-Viz side? It feels counter-intuitive to me that we extract the code for others to reuse, but we are not reusing it ourselves. Could we find a model where our Kedro-Viz still depends on whatever we extract, so we can have a single source of truth for the flowchart code?

Thanks for the diagrams, I think they clarify your thinking. Could you take a screenshot of Miro and paste them in your comment, so open-source users can see them?

[rashidakanchwala]

as discussed in our meeting today:-

• We will create a new repo for the standalone flowchart and in the next meeting we will agree on what parts of kedro-viz will go in the standalone app besides the flowchart (e.g. minimap)
• Kedro-viz initially will only import the graph calculation functions from the standalone flowchart and will have it's own flowchart.js/draw.js . However, eventually we want to do modularize both the flowchart.js and the draw.js in the standalone flowchart so that they can easily be extended to not only integrate better with Kedro-viz but also other apps (e.g. layerAI)

cc - @idanov , @tynandebold

[rashidakanchwala]

for the first iteration of the standalone component we are going to add the below functionality along with the svg -

• hover over nodes
• toolbar - dark/light, zoom, reset, zoom reference
• download to png/svg
• minimap

[yetudada]

I want to add the importance of bringing along the Brix and Layer team as primary users of this functionality. Let's communicate this plan to them and assess whether or not they will adopt this functionality. It would be remiss if we went through all of this effort and they did not use what we made.

Additionally, we should get user requirements for the additional components that need to be included with the flowchart from them. The Layer team also requires more than the flowchart.

At a high-level, Brix posts use:

[rashidakanchwala]

definitely, for now @ottis has shared the below; some of which we have already planned

• Can i render the DAG with a data structure like { nodes: [], edges: [] }
• Could we include a click handler for a node renderer that has access to the full node data?
• Zoom functionality could be another package you could export
• Minimap again could be a package as well
• Toolbar may not be super useful outside of kedro viz
• Export as png / svg could be a function export- UI doesn't need to be exported

[rashidakanchwala]

@yetudada @idanov -- we spoke to Brix team earlier this week. Below are updates:-

• They are happy with the way the flowchart currently works on Brix.
• They have a handful of users that use this feature so the adoption hasn't been much but it's also because they haven't really marketed this feature.

We shared our concern that the Kedro-viz within Brix had a smaller window (as it is embedded) and that it didn't give the user the full experience. To this they replied that they are already building a version on Brix that will open Kedro-viz in a new window and provide the full experience to the user.

Also @tynandebold was concerned that the if Brix might upgrade their Kedro-viz to the latest version it might break things due to incompatibility with React and he said he would look at fixing that.

Besides that there's no other additional work we need to do at the moment for Brix

[rashidakanchwala]

@ottis - You should be able to consume the layout algorithm from the latest Kedro-viz and it fixes the layout not working issue that you faced earlier.

When you install you might have to do npm i @quantumblack/kedro-viz --legacy-peer-deps to avoid any dependency clashes.

Also, here is an example of it implemented - https://tynandebold.com/kedro-viz-layout/, and here's the repo and file where it comes together: https://github.com/tynandebold/kedro-viz-layout/blob/main/src/App.js

cc- @tynandebold

I am closing this ticket for now. I will reopen if there is anything more to discuss.