Unable to start docker domjudge container on Synology (Btrfs): setfacl "Operation not supported"

[crdev]

Description of the problem

Domjudge docker container fails to start on Synology NFS due to numerous setfacl errors

Your environment

• DOMjudge version (e.g. 7.0.0 or a github commit hash): https://hub.docker.com/layers/domjudge/domserver/8.2.0/images/sha256-e9f74f05f9c3a9abb171cbdcd341a5283c230fab8edae426f2e5abf52b061e0d
• Operating system / Linux distribution and version (e.g. Ubuntu 18.04): DSM 7.1.1-42962 Update 4
• Webserver (e.g. Apache or nginx): As provided by the official docker image

Steps to reproduce

• Step 1: Create a docker stack on Synology (Btrfs) using the Yaml at the end of this report
• Step 2: Start the stack

Expected behaviour

The entire stack starts normally

Actual behaviour

After a few failures related to the not-yet-started MariaDB service, the following appears in the log and the container stops:

[..] Running start script 10-timezone.sh
Current default time zone: 'Europe/Amsterdam'
Local time is now:      Sat Mar 18 12:54:24 CET 2023.
Universal Time is now:  Sat Mar 18 11:54:24 UTC 2023.
[ok] Container timezone set to: Europe/Amsterdam
[..] Running start script 50-domjudge.sh
[..] Generating credential files
Password file 'dbpasswords.secret' already exists, leaving untouched.
Password file 'restapi.secret' already exists, leaving untouched.
Password file 'symfony_app.secret' already exists, leaving untouched.
Password file 'initial_admin_password.secret' already exists, leaving untouched.
Fix WEBAPP_BASEURL  -> /
[ok] Generated credential files
[..] Checking database connection
  Database installed; upgrading...
Database credentials read from '/opt/domjudge/domserver/etc/dbpasswords.secret'.
 [OK] Already at the latest version ("DoctrineMigrations\Version20230122162836")
setfacl: /opt/domjudge/domserver/webapp/var: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/jms_serializer_default: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/ContainerHg4GX85: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/pools: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/pools/system: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/pools/system/U072uNjGU7: Operation not supported
...
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/doctrine/orm: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/cache/prod/doctrine/orm/Proxies: Operation not supported
setfacl: /opt/domjudge/domserver/webapp/var/log: Operation not supported
[!!] Start script 50-domjudge.sh failed

Any other information that you want to share?

docker-compose.yml is as follows:

version: '3.9'

networks:
  domjudge:
    name: domjudge

services:
  mariadb:
    container_name: mariadb
    image: mariadb:latest
    volumes:
      - /volume1/docker/domjudge/mariadb:/var/lib/mysql
    networks:
      - domjudge
    ports:
      - 3306:3306
    environment:
      - MYSQL_ROOT_PASSWORD=<password>
      - MYSQL_USER=domjudge
      - MYSQL_PASSWORD=<password>
      - MYSQL_DATABASE=domjudge
    command: --max-connections=100

  domserver:
    container_name: domserver
    image: domjudge/domserver:latest
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    networks:
      - domjudge
    ports:
      - 12345:80
    depends_on:
      - mariadb
    environment:
      - CONTAINER_TIMEZONE=Europe/Amsterdam
      - MYSQL_HOST=mariadb
      - MYSQL_ROOT_PASSWORD=<password>
      - MYSQL_USER=domjudge
      - MYSQL_PASSWORD=<password>
      - MYSQL_DATABASE=domjudge

  judgehost-0:
    container_name: judgehost-0
    image: domjudge/judgehost:latest
    privileged: true
    hostname: judgedaemon-0
    volumes:
      - /sys/fs/cgroup:/sys/fs/cgroup:ro
    networks:
      - domjudge
    depends_on:
      - domserver
    environment:
      - DAEMON_ID=0
      - JUDGEDAEMON_PASSWORD=<password>

[nickygerritsen]

From Googling, I found this: https://github.com/elkarbackup/elkarbackup/issues/581

I stumbled into this problem when installing Elkarbackup on a QNAP NAS. To get rid of the error, i had to enable "advanced folder permissions" fond in Control Panel -> Privilege -> Shared folders -> Enable Advanced Folder Permissions

Could you try something like that?

If that doesn't work, we could try something like this: https://github.com/elkarbackup/elkarbackup/pull/647

[crdev]

Thanks Nicky, I had found that, too, yet it didn't apply as (1) I didn't use any mapped volumes, (2) I'm running Synology and the settings look a bit different. I've now recreated the stack with a mapped volume for /opt/domjudge/domserver/webapp/var and enabled advanced folder permissions for the parent shared folder, yet it didn't fix the issue for me.

I might get to trying out https://github.com/elkarbackup/elkarbackup/pull/647 some day :) and will then report back here.

[eldering]

I guess we should document as a requirement for DOMjudge that the underlying filesystem supports ACLs. BTRFS does have them enabled by default (see https://btrfs.readthedocs.io/en/latest/btrfs-man5.html), but maybe not on a Synology machine? Can you confirm that by running mount and checking the mount options (see if there's either acl or noacl mentioned) for the filesystem on which /opt/domjudge/domserver/webapp/var resides`?

[crdev]

Interesting, here is the output I got:

$ mount | grep acl
/dev/mapper/cachedev_0 on /volume1 type btrfs (rw,nodev,relatime,ssd,synoacl,nospace_cache,auto_reclaim_space,metadata_ratio=50,syno_allocator,subvolid=257,subvol=/@syno)
/dev/mapper/cachedev_0 on /volume1/@docker/btrfs type btrfs (rw,nodev,relatime,ssd,synoacl,nospace_cache,auto_reclaim_space,metadata_ratio=50,syno_allocator,subvolid=257,subvol=/@syno/@docker/btrfs)

Obviously, mentions synoacl...

[eldering]

That sounds like a Synology specific extension. Can you edit the mount options, e.g. in /etc/fstab and remove this?

[crdev]

Hmm, sadly, that's a mount for the master volume in /etc/fstab and it looks like:

/dev/mapper/cachedev_0 /volume1 btrfs auto_reclaim_space,ssd,synoacl,relatime,nodev 0 0,

which I'm not quite comfortable with changing settings for...

[crdev]

Related issue: https://github.com/DOMjudge/domjudge/issues/558

[ankon]

FWIW: "synoacl" is module from Synology indeed, you can find the sources in the archive kmsynoacl.txz at https://archive.synology.com/download/ToolChain/Synology%20NAS%20GPL%20Source/7.1.1-42962.

From the docs inside it:

MODULE_LICENSE("GPL");
MODULE_AUTHOR("Synology Inc.");
MODULE_DESCRIPTION("Add Windows ACL System Call Support");

... so this might be quite unrelated to btrfs acls :)

[vmcj]

I'm going to close this as we can't debug this without a Synology.

If this is indeed related to Synology following the NTFS ACL model there is no easy way for us to support both for this.