RPI aarch64: docker-compose is crashing at start

Open mrks1337 opened this issue 1 year ago • 5 comments

Hi, I am running the following docker-compose.yml on my Raspberry Pi (aarch64):

version: '3.9'

x-logging:
  &default-logging
  driver: json-file
  options:
    max-size: "100m"
    max-file: "3"

services:
  gitlab:
    image: zengxs/gitlab:ee-arm64
    container_name: gitlab
    restart: unless-stopped
    ports:
      - "9002:80"
      - "2222:22"
    environment:
      TZ: Europe/Berlin
      GITLAB_OMNIBUS_CONFIG: from_file('/home/pi/gitlab/gitlab.rb')
    volumes:
      # Your gitlab configurations file
      - ./gitlab.rb:/etc/gitlab.rb:ro
      # GitLab data dirs
      - /mnt/sda/gitlab:/etc/gitlab
      - /mnt/sda/gitlab:/var/log/gitlab
      - /mnt/sda/gitlab:/var/opt/gitlab
    logging: *default-logging

My gitlab.rb is in the same location as the docker-compose.yml (/home/pi/gitlab):

########## GitLab Configurations ##########
# More configurations see: https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-config-template/gitlab.rb.template

external_url "https://git.somedomain.com"
gitlab_rails['initial_root_password'] = somepass

When I check for my container, I only see this:

pi@raspberry:~/gitlab $ docker container ls | grep gitlab 39a8c090bc0b zengxs/gitlab:ee-arm64 "/assets/wrapper" 3 minutes ago Restarting (159) 56 seconds ago > gitlab

Did I miss something to configure?

May 25 '24 19:05 mrks1337

I noticed that your gitlab.rb configuration file is mounted at /etc/gitlab.rb inside the container. In this case, the GITLAB_OMNIBUS_CONFIG should also be configured as from_file('/etc/gitlab.rb').

Additionally, I noticed that your GitLab data directories all point to the same directory, which might also cause errors. I suggest you try the following configuration:

volumes:
  - /mnt/sda/gitlab-etc:/etc/gitlab
  - /mnt/sda/gitlab-logs:/var/log/gitlab
  - /mnt/sda/gitlab-data:/var/opt/gitlab

From my brief review, these seem to be the only two issues I've identified. Please try making these changes, and if it still doesn't work, let me know, and I'll take another look to see if there might be other issues.

May 27 '24 02:05 zengxs

Thank you for your quick reply!

I took over the changes that you recommend but I still get the same status that the container does not show any logs, nor starts normally.

Is there a difference between Rasp OS Bullseye and the new-ish bookworm version? Other than that I do now know how to investigate this problem any further without proper logs - do you?

May 27 '24 12:05 mrks1337

I’m very sorry, but I haven’t used Raspberry Pi and don’t have much experience with it.

However, could you try running a different Docker image to see if it works correctly? For instance, you could test with the hello-world image:

docker run hello-world

Also, to further investigate, you can check the system logs for any Docker-related entries. Try the following commands to gather more information:

sudo dmesg | grep docker
sudo journalctl -u docker.service

Let me know if this provides any clues or if the issue persists with different images.

Good luck!

May 28 '24 02:05 zengxs

I am finally able to get the following error shown during startup (upgraded to latest rasp OS and reinstalled docker with buildx):

pi@pi-docker:~/dev/gitlab $ docker-compose up -d Starting gitlab ... error

ERROR: for gitlab Cannot start service gitlab: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/pi/dev/gitlab/gitlab.rb" to rootfs at "/etc/gitlab.rb": mount /home/pi/dev/gitlab/gitlab.rb:/etc/gitlab.rb (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type

ERROR: for gitlab Cannot start service gitlab: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: error during container init: error mounting "/home/pi/dev/gitlab/gitlab.rb" to rootfs at "/etc/gitlab.rb": mount /home/pi/dev/gitlab/gitlab.rb:/etc/gitlab.rb (via /proc/self/fd/6), flags: 0x5001: not a directory: unknown: Are you trying to mount a directory onto a file (or vice-versa)? Check if the specified host path exists and is the expected type ERROR: Encountered errors while bringing up the project.

I am now unsure regarding the gitlab.rb file. Its located at /home/pi/dev/gitlab where also the docker-compose.yml and the gitlab.rb file is. Is it still correct that in the docker file, its configured as GITLAB_OMNIBUS_CONFIG: from_file('/home/pi/gitlab/gitlab.rb')?

When I have created the gitlab.rb, I get this error. When I do not create a file, I get those logs shown and it automatically creates a directory named gitlab.rb:

================================================================================ Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb

Errno::EISDIR

Is a directory @ io_fread - /etc/gitlab.rb

Cookbook Trace: (most recent call first)

/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:105:in block in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:35:in auto_vivify' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:105:in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:105:in block in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/config_mash.rb:35:in auto_vivify' /opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:105:in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:28:in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:27:in from_file' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab-ee/recipes/default.rb:22:in `from_file'

Relevant File Content:

/opt/gitlab/embedded/cookbooks/cache/cookbooks/package/libraries/settings_dsl.rb:

98: 99: def from_file(_file_path) 100: # Throw errors for unrecognized top level calls (usually spelling mistakes) 101: config_strict_mode true 102: # Turn on node deprecation messages 103: Gitlab::Deprecations::NodeAttribute.log_deprecations = true 104: # Allow auto mash creation during from_file call 105>> Gitlab::ConfigMash.auto_vivify { super } 106: ensure 107: config_strict_mode false 108: Gitlab::Deprecations::NodeAttribute.log_deprecations = false 109: end 110: 111: # Enhance set so strict mode errors aren't thrown as long as the setting is witin our defined config 112: def internal_set(symbol, value) 113: if configuration.key?(symbol) 114: configuration[symbol] = value

System Info:

chef_version=18.3.0 platform=ubuntu platform_version=22.04 ruby=ruby 3.1.5p253 (2024-04-023 revision 1945f8dc0e) [aarch64-linux] program_name=/opt/gitlab/embedded/bin/cinc-client executable=/opt/gitlab/embedded/bin/cinc-client

Running handlers: [2024-05-28T21:19:14+02:00] ERROR: Running exception handlers There was an error running gitlab-ctl reconfigure:

Is a directory @ io_fread - /etc/gitlab.rb

Running handlers complete [2024-05-28T21:19:14+02:00] ERROR: Exception handlers complete Infra Phase failed. 0 resources updated in 07 seconds [2024-05-28T21:19:14+02:00] FATAL: Stacktrace dumped to /opt/gitlab/embedded/cookbooks/cache/cinc-stacktrace.out [2024-05-28T21:19:14+02:00] FATAL: --------------------------------------------------------------------------------------- [2024-05-28T21:19:14+02:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT [2024-05-28T21:19:14+02:00] FATAL: --------------------------------------------------------------------------------------- [2024-05-28T21:19:14+02:00] FATAL: Errno::EISDIR: Is a directory @ io_fread - /etc/gitlab.rb

May 28 '24 19:05 mrks1337

Sorry for the delayed response.

In your docker-compose.yml file:

  volumes:
      - ./gitlab.rb:/etc/gitlab.rb:ro

The volumes configuration uses : as a delimiter. The part before the colon is the location of your file on the host system (i.e., the Raspberry Pi operating system). If it is a relative path, it should be relative to the directory where the docker-compose.yml file is located. The part after the colon is the location where the file will be mounted inside the container. There is also an additional :ro which indicates a read-only mount. With this parameter set, the container will only be able to read this file but not modify it.

The GITLAB_OMNIBUS_CONFIG configuration will be the path from which the configuration file is read inside the container. Therefore, it should point to the path after the colon, i.e., /etc/gitlab.rb.

If the corresponding file does not exist on your host system, Docker will default to mounting it as a directory and will create this directory. Therefore, you must create this gitlab.rb file in advance so that Docker can correctly recognize it as a file and not a directory.

In summary, you need to place gitlab.rb in the corresponding location in advance, and then modify the value of GITLAB_OMNIBUS_CONFIG to from_file('/etc/gitlab.rb').

Of course, if all of this still doesn't work, there is another solution, which is to avoid using a separate gitlab.rb file and instead write the configuration directly into the docker-compose.yml file.

You can configure GITLAB_OMNIBUS_CONFIG as follows:

environment:
  GITLAB_OMNIBUS_CONFIG: |
    # Copy all the content from your gitlab.rb here. Note the `|` symbol at the end of the previous line, and ensure the content below adheres to YAML indentation.
    external_url "https://git.somedomain.com"
    gitlab_rails['initial_root_password'] = somepass

If you still encounter issues, feel free to ask further questions. I hope you succeed in deploying this time.

May 30 '24 15:05 zengxs