gameserveer unexpectedly restarts all services

[aimuz]

What happened:

Without doing any updates, all my gs are restarted and the age is the same

➜  ~ kubectl -n pg-core-test get gs
NAME                        STATE   ADDRESS       PORT   NODE       AGE
battle-server-8kq7r-47wqv   Ready   10.77.0.215   7749   worker03   4m48s
battle-server-8kq7r-487dd   Ready   10.77.0.215   7158   worker03   5m17s
battle-server-8kq7r-9xzch   Ready   10.77.0.215   7173   worker03   5m17s
battle-server-8kq7r-qp2xr   Ready   10.77.0.215   7091   worker03   5m17s
battle-server-8kq7r-wp229   Ready   10.77.0.215   7524   worker03   5m17s
➜  ~ kubectl -n pg-core get gs
NAME                        STATE   ADDRESS       PORT   NODE       AGE
battle-server-mlgmp-c7m7f   Ready   10.77.0.215   7933   worker03   5m12s
battle-server-mlgmp-f5f2n   Ready   10.77.0.215   7049   worker03   5m20s
battle-server-mlgmp-tg4wl   Ready   10.77.0.215   7824   worker03   5m20s
battle-server-mlgmp-vcsqq   Ready   10.77.0.215   7720   worker03   5m20s
battle-server-mlgmp-xfwss   Ready   10.77.0.215   7196   worker03   5m20s
➜  ~

➜  ~ kubectl -n pg-core get events | grep Shutdown
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-25sdv   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-25sdv   Deleting Pod battle-server-mlgmp-25sdv
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-57c97   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-57c97   Deleting Pod battle-server-mlgmp-57c97
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-77dbs   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-77dbs   Deleting Pod battle-server-mlgmp-77dbs
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-bwtrq   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-bwtrq   Deleting Pod battle-server-mlgmp-bwtrq
4m54s       Normal    Shutdown           gameserver/battle-server-mlgmp-cj9h6   Deletion started
4m54s       Normal    Shutdown           gameserver/battle-server-mlgmp-cj9h6   Deleting Pod battle-server-mlgmp-cj9h6
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-cptvf   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-cptvf   Deleting Pod battle-server-mlgmp-cptvf
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f2gqt   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f2gqt   Deleting Pod battle-server-mlgmp-f2gqt
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f6wnw   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-f6wnw   Deleting Pod battle-server-mlgmp-f6wnw
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-flh8q   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-flh8q   Deleting Pod battle-server-mlgmp-flh8q
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-hchzw   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-hchzw   Deleting Pod battle-server-mlgmp-hchzw
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-jl542   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-jl542   Deleting Pod battle-server-mlgmp-jl542
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-kw5c5   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-kw5c5   Deleting Pod battle-server-mlgmp-kw5c5
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-lv6pv   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-lv6pv   Deleting Pod battle-server-mlgmp-lv6pv
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-mxk8j   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-mxk8j   Deleting Pod battle-server-mlgmp-mxk8j
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-pvxkq   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-pvxkq   Deleting Pod battle-server-mlgmp-pvxkq
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-qbmsz   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-qbmsz   Deleting Pod battle-server-mlgmp-qbmsz
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-rljxn   Deletion started
50m         Normal    Shutdown           gameserver/battle-server-mlgmp-rljxn   Deleting Pod battle-server-mlgmp-rljxn
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-tcd98   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-tcd98   Deleting Pod battle-server-mlgmp-tcd98
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-tvhjw   Deletion started
5m2s        Normal    Shutdown           gameserver/battle-server-mlgmp-tvhjw   Deleting Pod battle-server-mlgmp-tvhjw
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-xc8rc   Deletion started
25m         Normal    Shutdown           gameserver/battle-server-mlgmp-xc8rc   Deleting Pod battle-server-mlgmp-xc8rc
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-zhzdx   Deletion started
15m         Normal    Shutdown           gameserver/battle-server-mlgmp-zhzdx   Deleting Pod battle-server-mlgmp-zhzdx

What you expected to happen:

Curious as to why this is happening, is this a bug or a feature?

How to reproduce it (as minimally and precisely as possible):

Anything else we need to know?:

Environment:

• Agones version: 1.30.0
• Kubernetes version (use kubectl version): v1.24.6
• Cloud provider or hardware configuration:
• Install method (yaml/helm): helm
• Troubleshooting guide log(s):
• Others:

[zmerlynn]

That's curious, for sure. And there's nothing that could have triggered the game server itself to call Shutdown() (simultaneously)?

[aimuz]

No call, He is concentrated around 10:00 Eastern time, around 2:00 UTC

[aimuz]

A new day reappeared, it seems that the problem appeared when I upgraded to 1.30, I forget what version I belonged to before

Details

➜  ~ date
Thu May 11 09:43:04 CST 2023
➜  ~ kubectl -n pg-core get gs
NAME                        STATE   ADDRESS       PORT   NODE       AGE
battle-server-mlgmp-4ljq5   Ready   10.77.0.215   7176   worker03   3m50s
battle-server-mlgmp-dq25q   Ready   10.77.0.215   7153   worker03   3m50s
battle-server-mlgmp-hxkzk   Ready   10.77.0.215   7094   worker03   3m50s
battle-server-mlgmp-rkl5s   Ready   10.77.0.215   7144   worker03   3m50s
battle-server-mlgmp-tdpk2   Ready   10.77.0.215   7023   worker03   3m50s
➜  ~
➜  ~
➜  ~
➜  ~
➜  ~ kubectl -n pg-core-test get gs
NAME                        STATE       ADDRESS       PORT   NODE       AGE
battle-server-8kq7r-75v95   Ready       10.77.0.215   7563   worker03   4m7s
battle-server-8kq7r-cswz7   Ready       10.77.0.215   7322   worker03   8s
battle-server-8kq7r-l7ddv   Ready       10.77.0.215   7691   worker03   4m8s
battle-server-8kq7r-mgqrg   Ready       10.77.0.215   7785   worker03   4m8s
battle-server-8kq7r-pc7jb   Ready       10.77.0.215   7316   worker03   4m8s
battle-server-8kq7r-tgrsl   Allocated   10.77.0.215   7922   worker03   4m8s

[markmandel]

Did a node do a Kubernetes version upgrade?

[aimuz]

As far as I know no operations have been done and the problem has not occurred in the last few days

[markmandel]

Probably too late, but would be interesting to see the Pod or Node events - that would probably give us a better idea on why the Pod was shutdown.

[aimuz]

When I find the problem again, I'll add the pod and node-related events

[github-actions[bot]]

'This issue is marked as Stale due to inactivity for more than 30 days. To avoid being marked as 'stale' please add 'awaiting-maintainer' label or add a comment. Thank you for your contributions '

[github-actions[bot]]

This issue is marked as obsolete due to inactivity for last 60 days. To avoid issue getting closed in next 30 days, please add a comment or add 'awaiting-maintainer' label. Thank you for your contributions