volcano icon indicating copy to clipboard operation
volcano copied to clipboard

scheduler update job failed

Open WulixuanS opened this issue 2 years ago • 4 comments

What happened: A large number of error logs

E0907 03:39:28.604985       1 job_updater.go:105] Failed to update job <video/pgvx20230907113107.701322940-098ad-45e77d6a-f3d6-42a2-9002-c3b9f7c4c506>: Operation cannot be fulfilled on podgroups.scheduling.volcano.sh "pgvx20230907113107.701322940-098ad-45e77d6a-f3d6-42a2-9002-c3b9f7c4c506": StorageError: invalid object, Code: 4, Key: /registry/scheduling.volcano.sh/podgroups/video/pgvx20230907113107.701322940-098ad-45e77d6a-f3d6-42a2-9002-c3b9f7c4c506, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 265232bc-3fe2-47cb-8335-0808e5a569a4, UID in object meta:
E0907 03:39:28.636600       1 job_updater.go:105] Failed to update job <video/gvx20230907113235.385478059-2deeb-c3fdcbe3-6f70-44ed-ae32-3833abbfc01f>: Operation cannot be fulfilled on podgroups.scheduling.volcano.sh "gvx20230907113235.385478059-2deeb-c3fdcbe3-6f70-44ed-ae32-3833abbfc01f": StorageError: invalid object, Code: 4, Key: /registry/scheduling.volcano.sh/podgroups/video/gvx20230907113235.385478059-2deeb-c3fdcbe3-6f70-44ed-ae32-3833abbfc01f, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 673133c8-ba6a-468c-820b-b2c540769575, UID in object meta:
E0907 03:39:28.698327       1 job_updater.go:105] Failed to update job <video/gvx20230907113230.237306638-fcab6-af772ecb-df01-4977-ac5c-d30d7275b365>: Operation cannot be fulfilled on podgroups.scheduling.volcano.sh "gvx20230907113230.237306638-fcab6-af772ecb-df01-4977-ac5c-d30d7275b365": StorageError: invalid object, Code: 4, Key: /registry/scheduling.volcano.sh/podgroups/video/gvx20230907113230.237306638-fcab6-af772ecb-df01-4977-ac5c-d30d7275b365, ResourceVersion: 0, AdditionalErrorMsg: Precondition failed: UID in precondition: 020110aa-deca-4503-a635-87eb75e09ad2, UID in object meta:

What you expected to happen:

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

  • Volcano Version: v1.7.0
  • Kubernetes version (use kubectl version): v1.20.15
  • Cloud provider or hardware configuration:
  • OS (e.g. from /etc/os-release):
  • Kernel (e.g. uname -a):
  • Install tools:
  • Others:

WulixuanS avatar Sep 07 '23 03:09 WulixuanS

/assign @Mufengzhe

william-wang avatar Sep 08 '23 06:09 william-wang

@william-wang: GitHub didn't allow me to assign the following users: Mufengzhe.

Note that only volcano-sh members, repo collaborators and people who have commented on this issue/PR can be assigned. Additionally, issues/PRs can only have 10 assignees at the same time. For more information please see the contributor guide

In response to this:

/assign @Mufengzhe

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

volcano-sh-bot avatar Sep 08 '23 06:09 volcano-sh-bot

I did not find the problem you described. Can you provide more information to help me reproduce this problem?

Mufengzhe avatar Sep 12 '23 03:09 Mufengzhe

I took a look at the code, and there's only one place that could be causing this error.

calvin0327 avatar Apr 25 '24 06:04 calvin0327