Error 'failed calling webhook' while creating a job after installation

[wangmingyi11]

After we installed volcano, we've verified the status of Volcano components using kubectl get all -n volcano-system. It seems ok. Then we followed the demo online and tried to start a job using kubectl apply -f /home/queue.yaml --v=9. It reported the error below:

 curl -k -v -XGET  -H "Accept: application/json" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" 'https://192.168.4.202:6443/apis/batch.volcano.sh/v1alpha1/namespaces/default/jobs/test-job'
I0606 16:13:47.934075   83732 round_trippers.go:443] GET **https://192.168.4.202:6443/apis/batch.volcano.sh/v1alpha1/namespaces/default/jobs/test-job** 404 Not Found in 1 milliseconds
I0606 16:13:47.934091   83732 round_trippers.go:449] Response Headers:
I0606 16:13:47.934098   83732 round_trippers.go:452]     Content-Type: application/json
I0606 16:13:47.934104   83732 round_trippers.go:452]     Content-Length: 228
I0606 16:13:47.934110   83732 round_trippers.go:452]     Date: Mon, 06 Jun 2022 08:13:47 GMT
I0606 16:13:47.934129   83732 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"jobs.batch.volcano.sh \"test-job\" not found","reason":"NotFound","details":{"name":"test-job","group":"batch.volcano.sh","kind":"jobs"},"code":404}
I0606 16:13:47.934443   83732 request.go:1017] Request Body: {"apiVersion":"batch.volcano.sh/v1alpha1","kind":"Job","metadata":{"annotations":{"kubectl.kubernetes.io/last-applied-configuration":"{\"apiVersion\":\"batch.volcano.sh/v1alpha1\",\"kind\":\"Job\",\"metadata\":{\"annotations\":{},\"name\":\"test-job\",\"namespace\":\"default\"},\"spec\":{\"maxRetry\":5,\"minAvailable\":3,\"plugins\":{\"env\":[],\"ssh\":[],\"svc\":[]},\"policies\":[{\"action\":\"RestartJob\",\"event\":\"PodEvicted\"}],\"queue\":\"default\",\"schedulerName\":\"volcano\",\"tasks\":[{\"name\":\"default-nginx\",\"replicas\":6,\"template\":{\"metadata\":{\"name\":\"web\"},\"spec\":{\"containers\":[{\"image\":\"nginx\",\"imagePullPolicy\":\"IfNotPresent\",\"name\":\"nginx\",\"resources\":{\"requests\":{\"cpu\":\"1\"}}}],\"restartPolicy\":\"OnFailure\"}}}]}}\n"},"name":"test-job","namespace":"default"},"spec":{"maxRetry":5,"minAvailable":3,"plugins":{"env":[],"ssh":[],"svc":[]},"policies":[{"action":"RestartJob","event":"PodEvicted"}],"queue":"default","schedulerName":"volcano","tasks":[{"name":"default-nginx","replicas":6,"template":{"metadata":{"name":"web"},"spec":{"containers":[{"image":"nginx","imagePullPolicy":"IfNotPresent","name":"nginx","resources":{"requests":{"cpu":"1"}}}],"restartPolicy":"OnFailure"}}}]}}
I0606 16:13:47.934489   83732 round_trippers.go:423] curl -k -v -XPOST  -H "Content-Type: application/json" -H "User-Agent: kubectl/v1.17.3 (linux/amd64) kubernetes/06ad960" -H "Accept: application/json" 'https://192.168.4.202:6443/apis/batch.volcano.sh/v1alpha1/namespaces/default/jobs'
I0606 16:13:57.936339   83732 round_trippers.go:443] POST https://192.168.4.202:6443/apis/batch.volcano.sh/v1alpha1/namespaces/default/jobs 500 Internal Server Error in 10001 milliseconds
I0606 16:13:57.936378   83732 round_trippers.go:449] Response Headers:
I0606 16:13:57.936388   83732 round_trippers.go:452]     Content-Type: application/json
I0606 16:13:57.936394   83732 round_trippers.go:452]     Content-Length: 503
I0606 16:13:57.936401   83732 round_trippers.go:452]     Date: Mon, 06 Jun 2022 08:13:57 GMT
I0606 16:13:57.936440   83732 request.go:1017] Response Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Internal error occurred: failed calling webhook \"mutatejob.volcano.sh\": Post https://volcano-admission-service.volcano-system.svc:443/jobs/mutate?timeout=10s: context deadline exceeded","reason":"InternalError","details":{"causes":[{"message":"failed calling webhook \"mutatejob.volcano.sh\": Post https://volcano-admission-service.volcano-system.svc:443/jobs/mutate?timeout=10s: context deadline exceeded"}]},"code":500}
I0606 16:13:57.936715   83732 helpers.go:203] server response object: [{
  "kind": "Status",
  "apiVersion": "v1",
  "metadata": {},
  "status": "Failure",
  "message": "error when creating \"/home/honey/vc/0.yaml\": Internal error occurred: failed calling webhook \"mutatejob.volcano.sh\": Post https://volcano-admission-service.volcano-system.svc:443/jobs/mutate?timeout=10s: context deadline exceeded",
  "reason": "InternalError",
  "details": {
    "causes": [
      {
        "message": "failed calling webhook \"mutatejob.volcano.sh\": Post https://volcano-admission-service.volcano-system.svc:443/jobs/mutate?timeout=10s: context deadline exceeded"
      }
    ]
  },
  "code": 500
}]
F0606 16:13:57.936756   83732 helpers.go:114] Error from server (InternalError): error when creating "/home/honey/vc/0.yaml": Internal error occurred: **failed calling webhook** "mutatejob.volcano.sh": Post https://volcano-admission-service.volcano-system.svc:443/jobs/mutate?timeout=10s: context deadline exceeded

How to deal with it?

[william-wang]

@wangmingyi11 Please add your environment information like Kubernetes version, Volcano version, yaml file?

[wangmingyi11]

Kubernetes version:

# kubectl version
Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13., Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:07:13Z", GoVersion:"go1.13., Compiler:"gc", Platform:"linux/amd64"}

Volcano version and yaml: https://raw.githubusercontent.com/volcano-sh/volcano/master/installer/volcano-development.yaml

What we did:

cat <<EOF | kubectl apply -f - --v=9
apiVersion: scheduling.volcano.sh/v1beta1
kind: Queue
metadata:
  name: test
spec:
  weight: 1
  reclaimable: false
  capability:
    cpu: 2
EOF

@william-wang

[stale[bot]]

Hello 👋 Looks like there was no activity on this issue for last 90 days. Do you mind updating us on the status? Is this still reproducible or needed? If yes, just comment on this PR or push a commit. Thanks! 🤗 If there will be no activity for 60 days, this issue will be closed (we can always reopen an issue if we need!).

[stale[bot]]

Closing for now as there was no activity for last 60 days after marked as stale, let us know if you need this to be reopened! 🤗

[hwdef]

Does this problem still exist?

[hwdef]

Please try version 1.17 which will be released on 11.19

[AYUSHMT]

Yes , it still exists for me , I am testing it with kubernetes version - Client Version: v1.28.2 Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3 Server Version: v1.28.6

I installed the latest version of Volcano with kubectl command mentioned in the repo. Need some help on this

[lowang-bh]

Maybe volcano-admission-service is not up. Please check admission log.