argo-server can not work with Kubernetes proxy api

[zychina]

Pre-requisites

• [X] I have double-checked my configuration
• [X] I can confirm the issues exists when I tested with :latest
• [X] I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

I want to connect argo API Server to manage workflow, but I do not like to expose public endpoint in cluster as it required to order an additional LoadBalancer.

One option is using Kubernetes proxy api to proxy request to Argo Server, which can leverage existing Kubernetes API Server LB endpoint.

1. Install argo 3.4.0, I can not install latest 3.4.1 as anther error. k get pod -n argo NAME READY STATUS RESTARTS AGE argo-server-59f9dd46fb-99x22 1/1 Running 0 30s workflow-controller-69b9cc5b5f-5hpwt 1/1 Running 0 29s
2. setup Kubernetes API credential and endpiont cat $KUBECONFIG |grep client-certificate-data | awk -F ' ' '{print $2}' |base64 -d > ./client-cert.pem cat $KUBECONFIG |grep client-key-data | awk -F ' ' '{print $2}' |base64 -d > ./client-key.pem APISERVER=cat $KUBECONFIG |grep server | awk -F ' ' '{print $2}' `
3. curl kube proxy api to proxy request to argo server, but 503 returned. curl --cert ./client-cert.pem --key ./client-key.pem -k "$APISERVER/api/v1/namespaces/argo/pods/argo-server-59f9dd46fb-99x22:2746/proxy/api/v1/archived-workflows" { "kind": "Status", "apiVersion": "v1", "metadata": {}, "status": "Failure", "message": "error trying to reach service: EOF", "reason": "ServiceUnavailable", "code": 503 }
4. Curl workflow-controller healthz success with same way curl --cert ./client-cert.pem --key ./client-key.pem -k "$APISERVER/api/v1/namespaces/argo/pods/workflow-controller-69b9cc5b5f-5hpwt:6060/proxy/healthz" ok

Please help to take a look. Thank you. BTW, curl -k https://localhost:2746/api/v1/archived-workflows is good, when "kubectl -n argo port-forward deployment/argo-server 2746:2746". But port-forward is not option for me, as I have to write code to call api directly without setup a local proxy by port-forward.

Version

3.4.0

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

It is not workflow problem.

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

[sarabala1979]

I think it is an enhancement. can you make sure Argoserver is ready?

[zychina]

yes, Argo Server is ready. I can do "curl https://localhost:2746/api/v1/workflows/default" successfully, when running "kubectl -n argo port-forward service/argo-server 2746:2746".