Gitlab not longer reachable after update to 14.10

[geeks-r-us]

Hi,

I recently upgrade my container to 14.10. image. But now I face the issue that the container does not work correctly.

When I try to access web interface I get 404.

Here is container log:

Loading /etc/docker-gitlab/runtime/env-defaults Initializing logdir... Initializing datadir... Container TimeZone -> Europe/Berlin Installing configuration templates... SSL Key, SSL Certificate and DHParam were not found. Assuming that the container is running behind a HTTPS enabled load balancer. SSL key and certificates for Registry were not found Assuming that the Registry is running behind a HTTPS enabled load balancer. Configuring gitlab... Configuring gitlab::database Configuring gitlab::redis Configuring gitlab::actioncable Configuring gitlab::secrets... Configuring gitlab::sidekiq... Configuring gitlab::gitaly... Configuring gitlab::monitoring... Configuring gitlab::gitlab-workhorse... Configuring gitlab::puma... Configuring gitlab::timezone... Configuring gitlab::rack_attack... Configuring gitlab::ci... Configuring gitlab::artifacts... Configuring gitlab::packages... Configuring gitlab::terraform_state... Configuring gitlab::lfs... Configuring gitlab::uploads... Configuring gitlab::mattermost... Configuring gitlab::project_features... Configuring gitlab::oauth... Configuring gitlab::ldap... Configuring gitlab::cron_jobs... Configuring gitlab::backups... Configuring gitlab::backups::schedule... Configuring gitlab::registry... Configuring gitlab::pages... Configuring gitlab::sentry... Configuring gitlab::content_security_policy... Configuring gitlab-shell... Configuring nginx... Configuring nginx::gitlab... Configuring nginx::gitlab::hsts... 2022-05-01 13:15:20,449 CRIT Supervisor is running as root. Privileges were not dropped because no user is specified in the config file. If you intend to run as root, you can set user=root in the config file to avoid this message. 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/cron.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/gitaly.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/gitlab-workhorse.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/groups.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/mail_room.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/nginx.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/puma.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/sidekiq.conf" during parsing 2022-05-01 13:15:20,449 INFO Included extra file "/etc/supervisor/conf.d/sshd.conf" during parsing 2022-05-01 13:15:20,451 INFO RPC interface 'supervisor' initialized 2022-05-01 13:15:20,451 CRIT Server 'unix_http_server' running without any HTTP authentication checking 2022-05-01 13:15:20,452 INFO supervisord started with pid 1 2022-05-01 13:15:21,454 INFO spawned: 'gitaly' with pid 712 2022-05-01 13:15:21,455 INFO spawned: 'puma' with pid 713 2022-05-01 13:15:21,456 INFO spawned: 'gitlab-workhorse' with pid 714 2022-05-01 13:15:21,457 INFO spawned: 'sidekiq' with pid 715 2022-05-01 13:15:21,458 INFO spawned: 'sshd' with pid 721 2022-05-01 13:15:21,460 INFO spawned: 'nginx' with pid 722 2022-05-01 13:15:21,461 INFO spawned: 'cron' with pid 725 2022-05-01 13:15:22,699 INFO success: gitaly entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-01 13:15:22,699 INFO success: puma entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-01 13:15:22,699 INFO success: gitlab-workhorse entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-01 13:15:22,699 INFO success: sidekiq entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-01 13:15:22,699 INFO success: sshd entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-01 13:15:22,699 INFO success: nginx entered RUNNING state, process has stayed up for > than 1 seconds (startsecs) 2022-05-01 13:15:22,699 INFO success: cron entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)

Which indicates that nginx is running.

What I see on network is a continuous traffic between gitlab container and redis container of 200-400 kb/s

Update and migration seems to be finished finished:

docker-compose run gitlab app:rake db:migrate:status

showed 'up' for all migrations.

docker-compose run gitlab app:rake db:migrate:status

showed following issues:

Checking GitLab subtasks ...

Checking GitLab Shell ...

GitLab Shell: ... GitLab Shell version >= 13.25.1 ? ... OK (13.25.1) Running /home/git/gitlab-shell/bin/check Internal API available: FAILED - Internal API unreachable gitlab-shell self-check failed Try fixing it: Make sure GitLab is running; Check the gitlab-shell configuration file: sudo -u git -H editor /home/git/gitlab-shell/config.yml Please fix the error above and rerun the checks.

Checking GitLab Shell ... Finished

Checking Gitaly ...

Gitaly: ... default ... FAIL: 14:failed to connect to all addresses. debug_error_string:{"created":"@1651404125.733064868","description":"Failed to pick subchannel","file":"src/core/ext/filters/client_channel/client_channel.cc","file_line":3093,"referenced_errors":[{"created":"@1651404125.733064203","description":"failed to connect to all addresses","file":"src/core/lib/transport/error_utils.cc","file_line":163,"grpc_status":14}]}

Checking Gitaly ... Finished

Checking Sidekiq ...

Sidekiq: ... Running? ... no Try fixing it: sudo -u git -H RAILS_ENV=production bin/background_jobs start For more information see: doc/install/installation.md in section "Install Init Script" see log/sidekiq.log for possible errors Please fix the error above and rerun the checks.

Checking Sidekiq ... Finished

Checking Incoming Email ...

Incoming Email: ... Reply by email is disabled in config/gitlab.yml

Checking Incoming Email ... Finished

Checking LDAP ...

LDAP: ... LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab App ...

Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... yes Systemd unit files or init script up-to-date? ... yes

Anybody has sugestions how to fix that?

[lyenliang]

Which version did you upgrade from?

If the version number between the old and new gitlab is too large, the upgrade might fail.

[mashb1t]

potential solution: https://github.com/sameersbn/docker-gitlab/issues/2634#issuecomment-1384588210