Zach Loafman

I saw this fail recently as well. Grabbing as a good first issue.

Re: https://github.com/googleforgames/agones/issues/2445#issuecomment-1228927983, the problem here (and I ran into it as well), is that `-count ` is still constrained by `-timeout`, as evidenced by: ``` INFO[2022-08-26 19:45:55.082] Waiting for states...

Breadcrumbs: In the case of https://github.com/googleforgames/agones/issues/2445#issue-1110631394 and https://github.com/googleforgames/agones/pull/2782#issuecomment-1298755291 both, we see this kind of odd pattern: ``` Step #23 - "e2e-stable": time="2022-11-01 15:59:22.597" level=info msg="checking final crash state" gs=game-serverxsdhc state=Unhealthy...

I think in both cases I can see this pattern (** for emphasis): ``` Step #23 - "e2e-stable": time="2022-11-01 15:56:08.199" level=info msg="successfully crashed. Moving on!" test=TestGameServerRestartBeforeReadyCrash Step #23 - "e2e-stable":...

I have not been able to repro locally yet (hundreds of runs over the afternoon) - this may take some logs analysis. I scoured the build logs using the hyper-advanced...

Another dupe: https://github.com/googleforgames/agones/pull/2790#issuecomment-1301529764

So, I've been doing some logs analysis on https://github.com/googleforgames/agones/pull/2782#issuecomment-1298755291. What I see is that we crash the container twice, and due to backoff, there's a pretty big gap in health...

I have a theory as to what's happening here. I'll distill it down to a time-sequence first, then show the logs. I _think_ what's happening is: | time | event...

| So if the Pod is currently out of sync, the Update call should fail, since there is a newer generation in K8s. Hmm. I _think_ that as long as...

Wait! Yes, the second screenshot in https://github.com/googleforgames/agones/issues/2445#issuecomment-1301540694 had exactly that error message: `pod and gameserver game-serverxsdhc data are out of sync, retrying`