Git repos stay in "Failed" state after a reboot of the / unreachable git server

[erSitzt]

After upgrading our GitLab instance and a short downtime, almost all git repos went into "failed" state and did not recover. Only after manual "Force Update" they switch back to "Active"

The update of the git server was started approx. 11:30 and as of 11:35 gitjob did not log any new entries for the watched repos

[MKlimuszka]

@erSitzt , Can you give us the steps you used to update your GitLab repo so we can try to reproduce it? Also, what version of fleet are you using?

[erSitzt]

Rancher : 2.6.3-patch1 Fleet : v0.3.8

My GitLab update was just a normal apt-get update thing... i guess the gitlab-ctl restart after the update caused this. After a restart GitLab takes a few seconds before it is ready again and answers with a 502

After 30 seconds in my last attempt i can login to GitLab again, but Git-Repos in fleet stay in the failed state.

[erSitzt]

In short... immediately after a gitlab-ctl restart most of the git-repos go into failed state and dont recover automatically

Maybe this is a special case because GitLab sends the 502 ?

[erSitzt]

And no change in state after 1h ... all back to "active" after a "force update" again...