[frontend] Address KFPv2 Run Comparison page scalability

[zpChris]

Environment

• How did you deploy Kubeflow Pipelines (KFP)?

Kubeflow Pipelines Standalone

• KFP version:

2.0.0-alpha.3

Steps to reproduce

1. Go to the "Runs" tab on KFP
2. Select multiple runs which have many artifacts (for instance, some over 100+), and click "Compare Runs"

Expected result

You should see the page load all of the artifacts efficiently and accurately. Instead, the page currently does not load the artifacts past 100 per run, and the loading time will likely be slower in relation to comparing artifacts with few to no runs.

Materials and Reference

The main issue is shown in the TODO below:

https://github.com/kubeflow/pipelines/blob/a7fddf000d5da254ee872574677a974e4ff4b4db/frontend/src/pages/CompareV2.tsx#L315-L316

However, there is also a broader issue of scalability; though we implemented scalable methods and designed the page to handle a large quantity of artifacts, we have not thoroughly tested this page against runs with 100s of artifacts. Although the design and loading states should be able to provide a nice UX, the scalability of this page is nevertheless an important question to answer.

Lastly, this is also related to the issue of fetching very large HTML and Markdown files in an efficient manner, as the KFPv2 Run Comparison page will need to do this repeatedly. The existing implementation and TODO for this is available here: https://github.com/kubeflow/pipelines/blob/a7fddf000d5da254ee872574677a974e4ff4b4db/frontend/src/components/viewers/MetricsVisualizations.tsx#L890

---

Impacted by this bug? Give it a 👍. We prioritise the issues with the most 👍.