sameersbn
Not enough PostgreSQL connections
I am posting here first to ask if other people even have this issue - since the last few updates, our GitLab has increasingly produced errors randomly, and we finally narrowed it down to exhausting the PostgreSQL max_connection default setting of 100. This is a tricky problem, because a git clone might fail, or a random page on your server may render wrong or show an error, but a reload/retry will usually fix it, so it's horrible to debug. You can check if this is happening to you with a command like:
docker exec -it gitlab-server grep "sorry, too many clients already" /home/git/gitlab/log/application_json.log
We have about 15-20 users (some inactive), so I would expect this to be happening to many people by now. If you run an install around that size, please report here if you are seeing any errors like that.
Assuming that multiple people report back that they are seeing this in their logs, I suggest raising max_connections to 150 or 200, which can be done in multiple ways, but since the Postgres-Repository is abandoned, the simplest way for this image would be to add:
command:
- '-c max_connections=200'
The required connections don't seem to directly scale with users; gitlab.com itself only uses 500 according to some discussions in issues, my guess is that it's more closely related to used functionality where each aspect of GitLab (such as issues, pipelines) in use is responsible for a few open connections. There were mentions in patch notes the past where raising the limit was suggested.
Monitoring this week showed (after raising the limit further) that we have continuous spikes around 203 connections - I know the GitLab configuration has three reserved for administrative tasks, but no idea where it itself would be limited to 200 connections. Raising the connections to 400 has thus far resolved all issues, but from the looks of it we can get along with 203 - don't want to cut it too close, so we are considering permanently setting it to ~220. We are using https://github.com/prometheus-community/postgres_exporter to monitor our GitLab instance.
After the August GitLab update, our connections have stopped peaking over 200, and have never exceeded 119 again. We are changing this repo's 100 to 150 as described above and recommend that to be done here too, the change in the first post is all that is needed.