"the object has been modified" errors seen during allocation

[Kalaiselvi84]

What happened:

The failed e2e test: TestGameServerAllocationDuringMultipleAllocationClients

What you expected to happen:

Pass all the time

How to reproduce it (as minimally and precisely as possible):

It fails sometimes in the CI

Anything else we need to know?:

VERBOSE:     gameserverallocation_test.go:1417: 
VERBOSE:         	Error Trace:	/go/src/agones.dev/agones/test/e2e/gameserverallocation_test.go:1417
VERBOSE:         	Error:      	Not equal: 
VERBOSE:         	            	expected: 100
VERBOSE:         	            	actual  : 99
VERBOSE:         	Test:       	TestGameServerAllocationDuringMultipleAllocationClients
VERBOSE: --- FAIL: TestGameServerAllocationDuringMultipleAllocationClients (38.74s)

Environment:

• Agones version: Dev
• Kubernetes version (use kubectl version): 1.28
• Cloud provider or hardware configuration: GKE
• Install method (yaml/helm): helm
• Troubleshooting guide log(s): https://console.cloud.google.com/cloud-build/builds/8c2fe09e-b77b-4b9b-9ef5-5a5670ec7794?project=258182270954

[markmandel]

More and more!

https://console.cloud.google.com/cloud-build/builds/6322a904-b37c-4bcc-8e04-5e29a8315c21?project=258182270954

This is killing me.

[zmerlynn]

https://console.cloud.google.com/cloud-build/builds/6322a904-b37c-4bcc-8e04-5e29a8315c21?project=258182270954

could not completed gsa1 allocation : an error on the server ("error updating allocated gameserver: Operation cannot be fulfilled on gameservers.agones.dev \"preferred-m8hsh-mzrd8-g682d\": the object has been modified; please apply your changes to the latest version and try again

This is the same conflict @gongmax sees in perf tests, and I assume it's because two allocators try to allocate the same GameServer but fail (and it's somehow not retried by the allocator).

[zmerlynn]

This is also mentioned in the load test README: https://github.com/googleforgames/agones/blob/c1320eedf6ea21cb8e5e6a5e12005881ba725a64/test/load/allocation/README.md?plain=1#L62-L64

[github-actions[bot]]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '