allow setting Kubernetes node affinity in KFP pipelines

[mikekaram]

Similar to https://github.com/elyra-ai/elyra/issues/2823 Is your feature request related to a problem? Please describe. It is common practice in Kubernetes to select where pods are running by providing a node affinity to pods that match a node label or field, thus making sure that the pod will be scheduled to specific nodes that meet specific requirements (e.g., belong to specific zones or regions, have federated non-shareable data, have custom HW that the vendor hasn't created any device plugins and admission controllers yet or other use cases).

Currently, there is no way to specify a Node affinity in Elyra Kubeflow pipelines, this means that the selection of the node for running Elyra pipelines can be done only with node tolerations, a cumbersome practice in certain use cases.

Describe the solution you'd like We can resolve this by allowing users to specify a list of "node affinities" in a similar way to the "node taints" (see PR https://github.com/elyra-ai/elyra/pull/2848).

We only need to allow the user to set the type of node affinity (requiredDuringSchedulingIgnoredDuringExecution or preferredDuringSchedulingIgnoredDuringExecution), (weight*), key, operator, and value(s). * Since only the preferredDuringSchedulingIgnoredDuringExecution type needs a weight for its definition, I think it's an implementation decision if the user is allowed to select this parameter as well.

Additional context See the Assigning Pods to Nodes k8s docs See kfp.dsl.BaseOp().add_affinity() for more information about adding affinity in KFP See the Node Affinity v1 core spec for details about how a node affinity is defined.

[lresende]

I understand this is common for Admins, but how do we expect a data scientist working on this to know how to set these properties? My humble opinion is that we should expose nobs (e.g. GPU count) and based on that we internally would have the capabilities to translate this to whatever is needed in the runtime-specific way.

Having said that, I will not oppose adding these if you provide a pull request.