[Feature] Support networking.k8s.io.IngressSpec as ingress config for ray head

[ashahab]

Search before asking

• [X] I had searched in the issues and found no similar feature requirement.

Description

Currently the ray head only accepts a boolean "EnableIngress" to enable ingress for the ray head service. However, this precludes users from passing in additional information such as a different path structure or TLS config.

Therefore the kuberay should accept IngressSpec as an input. If EnableIngress is on, and IngressSpec is provided, then kuberay should create and reconcile the ingress config provided with the kubernetes apiserver.

Use case

Custom ingress configs such as a different path structure e.g: /dashboards/ray-cluster or different host TLS config,

Related issues

No response

Are you willing to submit a PR?

• [X] Yes I am willing to submit a PR!

[Tom-Newton]

I've been struggling a lot to setup ingresses for the ray dashboard and I think this issue would go a long way towards solving my problem.

Other similar github issues for kuberay:

1. Seems very similar but has less :+1:s https://github.com/ray-project/kuberay/issues/1436
2. Specifically asking for nginx.ingress.kubernetes.io/rewrite-target to be configured out of the box https://github.com/ray-project/kuberay/issues/2326
3. An old issue specifically about annotations, that seems to have been resolved https://github.com/ray-project/kuberay/issues/648. To me the most interesting point here is "We do not recommend using built-in ingress support, and we provide other ingress solutions in ingress.md." https://github.com/ray-project/kuberay/issues/648#issuecomment-1344604081

There are more

The ingress problems I've run into:

1. By default there is no nginx.ingress.kubernetes.io/rewrite-target so traffic gets directed to the wrong endpoint of the dashboard.
2. If I add nginx.ingress.kubernetes.io/rewrite-target it seems to break other ingresses using the same ingress controller, unless the ingress has a host. Kuberay creates ingresses without a host argument, so this causes problems. This behaviour is quite unexpected to me and it makes me wonder if there is something wrong with how our nginx ingress controller is configured.

Why I care about the built in ingresses

We do not recommend using built-in ingress support, and we provide other ingress solutions in ingress.md. https://github.com/ray-project/kuberay/issues/648#issuecomment-1344604081

Is there an alternative that automatically creates ingresses whenever a RayCluster is created? As I understand it, an operator pattern is the ideal way to automate the creation of the required ingresses, so it seems a bit strange if the recomendation is not to use kuberay's built in ingresses.

Configuring ingresses at the helm chart level This is probably a follow up issue, but I think its worth mentioning. I would prefer to configure ingresses on the kuberay helm chart, rather than on each RayCluster we create.

An example of this is the spark operator. It allows the host to be configured at the helm chart level with controller.uiIngress.urlFormat. There is also an in progress feature to allow configuring annotations and tls at the helm level.

[Tom-Newton]

I'm going to look into contributing something like what this issue proposed.