Evaluate use of Kubeflow Pipelines SDK v2

[ptitzler]

Is your feature request related to a problem? Please describe. Evaluate whether an upgrade to the KFP SDK v2 provides benefits and should be performed in in Elyra 3.x.

[ptitzler]

SDK v2 is still in beta. Waiting for it to be classified as stable or an urgent need arises that requires a switch.

[ozen]

@ptitzler Does this mean components written using KF pipelines sdk v2 cannot be used in Elyra?

Repo activity suggests KubeFlow 1.6.0 will be released next month with KF Pipelines 2.0.0 which makes SDK v2 default. Currently KF v1.6.0-rc.1 installs KF Pipelines 2.0.0-alpha.3. In Pipelines v2, v1 mode is removed from the namespace: https://github.com/kubeflow/pipelines/issues/7238

[ptitzler]

Elyra uses the SDK to communicate with KF (create experiments, upload pipelines, etc) and compile pipelines. I don't believe any of those tasks would fail after an upgrade. However, the visual pipeline editor parses component specifications to render the properties are used to customize the behavior of pipeline nodes. I do anticipate that some things related to this might not work as expected, but @kiersten-stokes and I can sort those out. I'm temporarily earmarking this for Elyra 4.0, which is a bit out [1], but we might be able to deliver earlier, depending on when a stable KFP SDK is released.

[1] Elyra 4.0 introduces support for JupyterLab 4, which itself has a moving target date.