[Bug]: Ownership of all directory is overridden

[waki285]

Error Message and Logs

The following error message is displayed, and at this point any files' ownership is overridden to the non-root user (coolify operated user)

sudo: /etc/sudo.conf is owned by uid 1001, should be 0
sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set
sudo: /etc/sudo.conf is owned by uid 1001, should be 0
sudo: /usr/bin/sudo must be owned by uid 0 and have the setuid bit set

# ls -la /
total 88
drwxr-xr-x  23 ubuntu ubuntu  4096 Feb 12 15:47 .
drwxr-xr-x  23 ubuntu ubuntu  4096 Feb 12 15:47 ..
lrwxrwxrwx   1 ubuntu ubuntu     7 Apr 22  2024 bin -> usr/bin
drwxr-xr-x   2 ubuntu ubuntu  4096 Feb 26  2024 bin.usr-is-merged
drwxr-xr-x   5 ubuntu ubuntu  4096 Feb 13 06:47 boot
drwxr-xr-x   3 ubuntu ubuntu  4096 Feb 12 15:47 data
drwxr-xr-x  19 ubuntu ubuntu  3860 Feb 12 15:33 dev
drwxr-xr-x 115 ubuntu ubuntu  4096 Feb 16 15:38 etc
drwxr-xr-x   4 ubuntu ubuntu  4096 Feb 12 15:33 home
lrwxrwxrwx   1 ubuntu ubuntu     7 Apr 22  2024 lib -> usr/lib
drwxr-xr-x   2 ubuntu ubuntu  4096 Apr  8  2024 lib.usr-is-merged
drwx------   2 ubuntu ubuntu 16384 Oct  9 08:41 lost+found
drwxr-xr-x   2 ubuntu ubuntu  4096 Oct  9 08:38 media
drwxr-xr-x   2 ubuntu ubuntu  4096 Oct  9 08:38 mnt
drwxr-xr-x   3 ubuntu ubuntu  4096 Feb 12 15:47 opt
dr-xr-xr-x 231 ubuntu ubuntu     0 Feb 12 15:32 proc
drwx------   7 ubuntu ubuntu  4096 Feb 16 15:38 root
drwxr-xr-x  37 ubuntu ubuntu  1260 Feb 16 15:35 run
lrwxrwxrwx   1 ubuntu ubuntu     8 Apr 22  2024 sbin -> usr/sbin
drwxr-xr-x   2 ubuntu ubuntu  4096 Mar 31  2024 sbin.usr-is-merged
drwxr-xr-x   6 ubuntu ubuntu  4096 Oct  9 08:44 snap
drwxr-xr-x   2 ubuntu ubuntu  4096 Oct  9 08:38 srv
dr-xr-xr-x  13 ubuntu ubuntu     0 Feb 16 15:31 sys
drwxrwxrwt  12 ubuntu ubuntu  4096 Feb 16 15:40 tmp
drwxr-xr-x  11 ubuntu ubuntu  4096 Oct  9 08:38 usr
drwxr-xr-x  13 ubuntu ubuntu  4096 Feb 12 15:32 var

Steps to Reproduce

1. Use server of non-root user
2. create service with Docker Compose Empty
3. use this compose file (this is copied and modified from prom/node-exporter)

services:
  node_exporter:
    image: 'prom/node-exporter:latest'
    command:
      - '--path.rootfs=/host'
    network_mode: host
    pid: host
    restart: unless-stopped
    volumes:
      - '/:/host:ro,rslave'

4. once the container is created, try to change the name from the default name to something different (don't click Deploy)
5. error occurs

Example Repository URL

No response

Coolify Version

v4.0.0-beta.393

Are you using Coolify Cloud?

No (self-hosted)

Operating System and Version (self-hosted)

Ubuntu 24.04.2 LTS

Additional Information

I'm using OCI Instance

[shashank-sharma]

I faced the same issue twice, and it got triggered when I changed the name of the container, and the worst part is, recovery is very hard as I am not able to do su - or sudo. Is there a way to avoid this from happening?

[ReCREWt-Philipp]

Same problem here. Sudoers file ownership is set to my non-root user with no way of recovery! Please fix ASAP as this causes us a lot of problems!

[odraude7]

Hello everyone,

I encountered this permission issue where critical system files and directories had their ownership changed (resulting in errors like "sudo: /etc/sudo.conf is owned by uid 1003, should be 0" and "sudo: /etc/sudoers is owned by uid 1003, should be 0"). As a temporary workaround, I developed the following script—with help from ChatGPT—that resets the permissions and ownership of essential directories and files.

Instructions:

1. Log in as root (using su - or via recovery mode).
2. Save the script below in a file (for example, fix_permissions.sh).
3. Make the script executable:

   chmod +x fix_permissions.sh
   

4. Run the script:

   ./fix_permissions.sh
   

Script:

#!/bin/bash
# Script to fix critical system permissions, including sudo configuration files.
# Run this script as root (do not use sudo to run this script).

echo "Starting permission fixes..."

# 1. Fix ownership and permissions of the root directory and main directories
chown root:root /
chmod 755 /

# 2. Fix ownership of critical system directories (excluding /usr, /root, and /usr/local)
chown root:root /bin /boot /dev /etc /lib /lib64 /opt /proc /run /sbin /srv /sys /tmp /var /var/spool
chmod 755 /bin /boot /dev /etc /lib /lib64 /opt /proc /run /sbin /srv /sys /tmp /var /var/spool

# 3. Fix permissions for /tmp
chmod 1777 /tmp

# 4. Fix /usr and /usr/local directories (only the directory itself, non-recursively)
chown root:root /usr
chmod 755 /usr
if [ -d /usr/local ]; then
    chown root:root /usr/local
    chmod 755 /usr/local
fi

# 5. Fix /root (home for root)
chown root:root /root
chmod 700 /root

# 6. Interactive fix for directories in /home (for individual users)
if [ -d /home ]; then
    echo "Detected the following directories in /home:"
    for dir in /home/*/ ; do
        user=$(basename "$dir")
        read -p "Do you want to fix permissions for /home/$user so that it belongs to $user? (y/n) " resp
        if [[ "$resp" =~ ^[Yy]$ ]]; then
            chown -R "$user:$user" "/home/$user"
            echo "Permissions fixed for /home/$user"
        else
            echo "Skipping /home/$user"
        fi
    done
fi

# 7. Fix directories related to coolify and other altered directories
chown -R root:root /data /snap /srv
chmod 755 /data /snap /srv

# 8. Fix the sudo binary
chown root:root /usr/bin/sudo
chmod 4755 /usr/bin/sudo

# 9. Fix sudo configuration files
if [ -f /etc/sudo.conf ]; then
    chown root:root /etc/sudo.conf
    chmod 644 /etc/sudo.conf
fi

if [ -f /etc/sudoers ]; then
    chown root:root /etc/sudoers
    chmod 440 /etc/sudoers
fi

if [ -d /etc/sudoers.d ]; then
    chown -R root:root /etc/sudoers.d
    chmod 755 /etc/sudoers.d
    for file in /etc/sudoers.d/*; do
        if [ -f "$file" ]; then
            chown root:root "$file"
            chmod 440 "$file"
        fi
    done
fi

# 10. Fix various subdirectories under /var (cache, lib, etc.)
for dir in /var/backups /var/cache/apt /var/cache/cloud /var/cache/command-not-found /var/cache/logrotate \
           /var/cache/man-db /var/cache/PackageKit /var/cache/pam /var/cache/plymouth /var/cache/sgml-base \
           /var/cache/snapd /var/cache/sudo /var/cache/unattended-upgrades /var/cache/update-notifier /var/cache/vim \
           /var/lib/apt /var/lib/cloud /var/lib/command-not-found /var/lib/git /var/lib/grub /var/lib/logrotate \
           /var/lib/man-db /var/lib/PackageKit /var/lib/pam /var/lib/plymouth /var/lib/sgml-base /var/lib/snapd \
           /var/lib/sudo /var/lib/ubuntu-advantage /var/lib/ubuntu-drivers-common /var/lib/ubuntu-release-upgrader \
           /var/lib/ucf /var/lib/update-manager /var/lib/update-notifier /var/lib/vim; do
    if [ -d "$dir" ]; then
        chown -R root:root "$dir"
    fi
done

# 11. Adjust permissions for /var/lock (if it exists)
if [ -e /var/lock ]; then
    chown root:root /var/lock
    chmod 755 /var/lock
fi

echo "Permission fixes completed."

Notes:

• This script resets the ownership and permissions of the root directory, key system directories, /root, /usr, /usr/local, and also interactively fixes the home directories found in /home.
• It specifically corrects the sudo binary and configuration files (/etc/sudo.conf, /etc/sudoers, and files in /etc/sudoers.d) to resolve the reported errors.
• This is a temporary fix intended to restore system functionality while a more permanent solution is considered.

Hope this helps!

Cheers,
Edu (with help from ChatGPT)

[WhiskerLogic]

I also had this happen to me, using a non root user. Lots of system files and directory ownership was switched from root to the user coolify was using to ssh manage the server. I couldn't use any sudo commands because the directory ownership was altered on the server:

sudo: /usr/bin/sudo must be owed by uid 0 and have the setuid bit set

I had to setup a new server and migrate everything out. To get get sudo back you have to login as root and run the commands from there.

[ReCREWt-Philipp]

Update: After updating to v4.0.0-beta.404 this issue no longer appears for us! Let's hope it stays this way.

[heiniha]

Please, label this bug as critical, since it corrupted the file system permissions on one of our servers, and we had to restore it from backup.

The bug is triggered from a basic service rename inside the Coolify web app, which results in the Coolify manager going rogue and stealing permissions from root for a large part of the file system. After that, Coolify loses connection to the server, and the permissions become difficult to recover to their original state.

Coolify version: v4.0.0-beta.420.6 Ubuntu version: Ubuntu 24.04.3 LTS Kernel version: 6.8.0-71-generic

Steps:

1. Add a new "Docker Compose Empty" resource.
2. In the setup, paste in the following Docker Compose file from Loki's official documentation: https://raw.githubusercontent.com/grafana/loki/v3.4.1/production/docker-compose.yaml and click "Save".
3. On the service page, click "Edit Compose File" and remove all services except for Loki and click "Save".
4. Enter the service page and set "Service Name" to something different and click "Save".
5. Now the bug triggers and the connection to the underlying server is lost.
6. SSH into that server and verify that the non-root-coolify-user has stolen permissions from root for many essential files, leading to system corruption. For example, the files in /var/log and /var/lib.

[peaklabs-dev]

Just to be clear, we're talking about changing the name here?

[heiniha]

Just to be clear, we're talking about changing the name here?

Yes it is that field "Name", but when using "Docker Compose" it is instead called "Service Name".

[peaklabs-dev]

I tried reproducing this issue but I could not determine why it would happen, it is certainly not because of renaming or at least not there.

As non-root user support will only officially be supported (non-experimental) on v5, I will leave this open for now. If anyone finds where/why the error occurs, we can try to fix it ofc but I think this will be difficult as then probably other issues will occur since many places use hardcoded permissions and sometimes even hardcoded users.

[shrinidhi4u]

This is a serious bug that I faced today. It changed the permissions of my entire file system to ubuntu user from root. I lost complete sudo access on a production vm. I lost an entire day to just figure out what happened. I am surprised that this bug is still open. Below is the docker-compose file I had used (self-hosted Coolify v4.0.0-beta.420.6.)

version: "3.8"
services:
  node-exporter:
    image: prom/node-exporter:v1.8.2
    container_name: node-exporter
    restart: unless-stopped
    pid: host
    network_mode: host
    command: ["--path.rootfs=/host"]
    volumes:
      - /:/host:ro,rslave

  cadvisor:
    image: gcr.io/cadvisor/cadvisor:v0.49.1
    container_name: cadvisor
    restart: unless-stopped
    network_mode: host
    privileged: true
    command: ["--port=9080"]
    volumes:
      - /:/rootfs:ro
      - /var/run:/var/run:ro
      - /sys:/sys:ro
      - /var/lib/docker/:/var/lib/docker:ro
      - /dev/disk/:/dev/disk:ro

[shrinidhi4u]

@andrasbacsai @peaklabs-dev Hi folks, may I draw your attention to this serious bug?

[WhiskerLogic]

@shrinidhi4u

This is a serious bug that I faced today. It changed the permissions of my entire file system...

I has this exact same story.. I got locked out (ssh didn't work) because the sudo/ssh dir was messed up.

As a fix for now, I'm using root access for all my servers. This seems to work, rather than using a user with sudo privileges for coolify to ssh. For almost a year now I haven't had any file/folder permission issues.

[shrinidhi4u]

@WhiskerLogic Thanks for the comment. Unfortunately, the VPS I get from Oracle doesn't provide root user. I only get a user that has sudo privileges. Otherwise, I would have used root user.

[andrasbacsai]

I identified the error. Working on a solution.

[WhiskerLogic]

I identified the error. Working on a solution.

Thank you so much!!

[andrasbacsai]

A fix is coming in next version.