kuberay
kuberay copied to clipboard
ray-project
[Bug][Docs] Ingress EKS bug and documentation
Search before asking
- [X] I searched the issues and found no similar issues.
KubeRay Component
ray-operator
What happened + What you expected to happen
The KubeRay operator's ingress support seems not to work on EKS: https://github.com/ray-project/kuberay/blob/76f96bf5a05651611f9a8734c34c2a7f23f54b54/ray-operator/controllers/ray/common/ingress.go#L80
This issue tracks the linked TODO.
In general, ingress setup is sensitive to cloud provider context, so it's important to explain that it is fine not to enable automatic configuration of an ingress.
There are three tasks here: (P0) Clarify in the docs that it's not required to have KubeRay configure an ingress for you -- you can do it yourself if necessary. (Just set the service type as required.) (P1) Explain in greater detail what kind of Ingress the operator can set up and under what conditions it's suitable to use the ingress. (P2) Fix the EKS bug.
Reproduction script
Try to use enableIngress: true on EKS. (I haven't verified the bug myself.)
Anything else
No response
Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
The description looks good to me. Let me know the following and task assignment.
@brucez-anyscale I tagged you mostly because you identified the particular issue with EKS re: task assignment, feel free to claim it if you have time!
I can take it. Let me find some time to prepare a pr. Maybe later this week or next week.
Any progress? Meet same problem on k8s v1.23.6 Following doc https://ray-project.github.io/kuberay/guidance/ingress/ but can not create ingress. Error message : "can not be set when the class field is also set" k8s ValidateIngressCreate source code: https://github.com/kubernetes/kubernetes/blob/v1.23.6/pkg/apis/networking/validation/validation.go#L235
@DmitriGekhtman Could you help to follow up? Maybe @kevin85421 could help?
We've added some documentation on ingress setup to the KubeRay repo in https://github.com/ray-project/kuberay/pull/699 and https://github.com/ray-project/kuberay/pull/658. We should also make sure to explain ingress setup clearly in the Ray docs.
Basically, anywhere we currently mention port-forwarding in the KubeRay docs or in the Ray docs, we should explain that port-forwarding is a hack for simplicity of the example. We should link somewhere where ingress setup is explained. cc @kevin85421 @sihanwang41 @waleedkadous
@DmitriGekhtman Can we close this issue? We did not recommend any user use built-in ingress support.
The last to-do is to remove documentation concerning built-in ingress support.
Also, you need to either explain how to do in the Ray documentation or link to KubeRay documentation.
Anywhere in the Ray docs where we tell the user to port-forward the dashboard port, it needs to say that port-forwarding is not appropriate for production and should link to ingress setup docs.
Also, we need to make it very explicit that users should not use an ingress to expose Ray client -- a better workflow for Ray client is to set up a Jupyter notebook in the same K8s cluster, connect with Ray client from the Jupyter environment, and expose the Jupyter server with an ingress.
Broader item for improvement of the Ray docs: https://github.com/ray-project/kuberay/issues/828
The last to-do is to remove documentation concerning built-in ingress support.
Done (https://github.com/ray-project/kuberay/pull/999)
Close this one, and track the progress in #828.