Slow loading times / bad performance on a fresh local install

[christianvw]

Hi,

I am experiencing overall slow performance on my gitlab installation, even though it is locally running.

Are there any performance tweaks to make?

This should not be an hardware issue, since I use a Ryzen 5 1600X (6 cores, 12 threads) with 16 GB DDR4 3200MHz RAM and a SATA SSD.

Slow performance means:

• An initial page load in web ui (for example of a repository) takes multiple seconds, this can be seen in the network tab of the browser (the initial request takes most of the time, at least 2 full seconds; the DOM is then finally ready in more than 3 full seconds)
• It could be the API response time which is slow, since curl -I http://localhost/it/testproject returns X-Runtime: 2.230676 for example
• The slow response time disappears when the exact same repository has been viewed a few times in web UI. Then the intial request takes only 263ms. This sounds like caching.
• Most affected are the repository pages in web ui (for example http://localhost/it/textproject).
• Also after a container restart the performance at start is verly slow.

What I did already:

• Increased the PUMA_WORKERS to 5 (cpu cores - 1), from https://docs.gitlab.com/ee/install/requirements.html#puma-workers
• Set the puma threads to 4 (PUMA_THREADS_MIN and PUMA_THREADS_MAX), from https://docs.gitlab.com/ee/install/requirements.html#puma-threads
• Increased the maximum ram a puma worker can get until it gets restarted to 1250 megabytes (PUMA_PER_WORKER_MAX_MEMORY_MB)

My docker-compose.yml looks like the example (https://github.com/sameersbn/docker-gitlab/blob/master/docker-compose.yml) except the environment variables described above.

Content of my docker-compose.yml

version: '2.3'

services:
  production-gitlab-redis:
    container_name: production-gitlab-redis
    image: redis:5.0.9
    command:
      - --loglevel warning
    volumes:
      - ./redis:/var/lib/redis:Z

  production-gitlab-postgresql:
    container_name: production-gitlab-postgresql
    image: sameersbn/postgresql:12-20200524
    volumes:
      - ./postgresql:/var/lib/postgresql:Z
    environment:
      - DB_USER=gitlab
      - DB_PASS=password
      - DB_NAME=gitlab
      - DB_EXTENSION=pg_trgm,btree_gist

  production-gitlab-gitlab:
    container_name: production-gitlab-website
    image: sameersbn/gitlab:13.7.1
    depends_on:
      - production-gitlab-redis
      - production-gitlab-postgresql
    ports:
      - "80:80"
      - "26:22"
    volumes:
      - ./gitlab:/home/git/data:Z
    healthcheck:
      test: ["CMD", "/usr/local/sbin/healthcheck"]
      interval: 5m
      timeout: 10s
      retries: 3
      start_period: 5m
    environment:
    - DEBUG=false

    - DB_ADAPTER=postgresql
    - DB_HOST=production-gitlab-postgresql
    - DB_PORT=5432
    - DB_USER=gitlab
    - DB_PASS=password
    - DB_NAME=gitlab

    - REDIS_HOST=production-gitlab-redis
    - REDIS_PORT=6379

    - TZ=Europe/Berlin
    - GITLAB_TIMEZONE=Berlin

    - GITLAB_HTTPS=false
    - SSL_SELF_SIGNED=false

    - GITLAB_HOST=localhost
    - GITLAB_PORT=80
    - GITLAB_SSH_PORT=26
    - GITLAB_RELATIVE_URL_ROOT=
    
    - PUMA_WORKERS=5
    - PUMA_THREADS_MIN=4
    - PUMA_THREADS_MAX=4
    - PUMA_PER_WORKER_MAX_MEMORY_MB=1250
    
    - GITLAB_SECRETS_DB_KEY_BASE=secret
    - GITLAB_SECRETS_SECRET_KEY_BASE=secret
    - GITLAB_SECRETS_OTP_KEY_BASE=secret

    - GITLAB_NOTIFY_ON_BROKEN_BUILDS=false
    - GITLAB_NOTIFY_PUSHER=false

Are there any performance tweaks I could do? I would like to get the response time to < 1 second, like gitlab.com did.

[tomastzn]

I started to see this issue recently on my installation as well, several views are now taking seconds to load. It didn't use to be this slow.

[SerdarNur]

@christianvw @tomastzn i have the same issue .. we set it up before a few month and it was very fast. Now with some projects and users its taking seconds to load .. any solutions?

[Glideh]

I just did a fresh 14.1.3 install and I'm getting random 1/2/3s response times.

Here is an example from production.log claiming the views took more than 8s to render

Started GET "/admin/application_settings/general" for 192.168.125.254 at 2021-08-24 17:24:24 +0200 Processing by Admin::ApplicationSettingsController#general as HTML Rendered layout layouts/admin.html.haml (Duration: 8455.6ms | Allocations: 341159) Completed 200 OK in 8543ms (Views: 8446.1ms | ActiveRecord: 21.5ms | Elasticsearch: 0.0ms | Allocations: 348520)

Processes are not taking so much CPU/Memory although I'm seeing Ruby constantly running around 3% CPU average (maybe the healthcheck).

Here is my composition

version: '2.3'

services:
  redis:
    restart: unless-stopped
    image: redis:6.2
    command:
    - --loglevel warning
    volumes:
    - redis-data:/data:Z

  db:
    restart: unless-stopped
    image: sameersbn/postgresql:12-20200524
    volumes:
    - db-data:/var/lib/postgresql:Z
    environment:
    - DB_USER=gitlab
    - DB_PASS=${DB_PASS}
    - DB_NAME=gitlab
    - DB_EXTENSION=pg_trgm,btree_gist

  gitlab:
    restart: unless-stopped
    image: sameersbn/gitlab:14.1.3
    depends_on:
    - redis
    - db
    ports:
    - "80:80"
    - "4446:22"
    volumes:
    - gitlab-data:/home/git/data:Z
    - ./log:/home/git/gitlab/log
    healthcheck:
      test: ["CMD", "/usr/local/sbin/healthcheck"]
      interval: 5m
      timeout: 10s
      retries: 3
      start_period: 5m
    environment:
    - DEBUG=false

    - DB_HOST=db
    - DB_USER=gitlab
    - DB_PASS=${DB_PASS}
    - DB_NAME=gitlab

    - REDIS_HOST=redis

    - TZ=Europe/Paris
    - GITLAB_TIMEZONE=Paris

    - GITLAB_HOST=gitlab.acme.fr
    - GITLAB_SECRETS_DB_KEY_BASE=<random>
    - GITLAB_SECRETS_SECRET_KEY_BASE=<random2>
    - GITLAB_SECRETS_OTP_KEY_BASE=<random3>

    - GITLAB_ROOT_PASSWORD=${GITLAB_ROOT_PASS}
    - GITLAB_ROOT_EMAIL=${GITLAB_ROOT_EMAIL}

#    - [email protected]
#    - [email protected]

    - GITLAB_BACKUP_SCHEDULE=daily
    - GITLAB_BACKUP_TIME=01:00

#    - SMTP_ENABLED=false
#    - SMTP_DOMAIN=www.example.com
#    - SMTP_HOST=smtp.gmail.com
#    - SMTP_PORT=587
#    - [email protected]
#    - SMTP_PASS=password
#    - SMTP_STARTTLS=true
#    - SMTP_AUTHENTICATION=login

volumes:
  redis-data:
  db-data:
  gitlab-data:

[stephane-klein]

@christianvw @Glideh have you some news about your performance issue?

[stephane-klein]

I have also performance issues, for instance:

Started GET "/foo/bar/-/issues/6401" for x.x.x.x at 2022-05-21 18:15:09 +0200
Processing by Projects::IssuesController#show as HTML
  Parameters: {"namespace_id"=>"foo", "project_id"=>"bar", "id"=>"6401"}
Completed 200 OK in 1635ms (Views: 1225.3ms | ActiveRecord: 155.7ms | Elasticsearch: 0.0ms | Allocations: 1064074)

I run GitLab 14.10.2 on this server:

• Processor 1× AMD EPYC 7281 CPU - 16C/32T - 2.1 GHz
• 96 GB DDR4 ECC

I set this parameter in docker-compose.yml:

  postgres:
    image: index.docker.io/postgres:12.4-alpine
    command: >
      -c shared_buffers=2000MB
      -c maintenance_work_mem=150MB
      -c work_mem=50MB
      -c seq_page_cost=0.8
      -c random_page_cost=1.1
      -c effective_io_concurrency=200
    ...

  gitlab:
    ...
    environment:

      ...
      # 32*16
      DB_POOL: 512

      ...

      PUMA_WORKERS: 32
      PUMA_THREADS_MIN: 1
      # https://docs.gitlab.com/ee/install/requirements.html#puma-threads
      PUMA_THREADS_MAX: 4
      PUMA_PER_WORKER_MAX_MEMORY_MB: 4048
      PUMA_MASTER_MAX_MEMORY_MB: 20048