[Heartbeat] permission denied error when running on Azure Container Instances (ACI)

[zmoog]

For confirmed bugs, please report:

• Version:
• Operating System:
• Discuss Forum URL:
• Steps to Reproduce:

Customer was running heartbeat -environment container --strict.perms=false --path.config /mnt/secrets in Azure Container Instances (ACI) service for months without issues, but recently it stopped working due to:

2022-09-15T12:34:25.257Z INFO instance/beat.go:686 Home path: [/usr/share/heartbeat] Config path: [/mnt/secrets] Data path: [/usr/share/heartbeat/data] Logs path: [/usr/share/heartbeat/logs] Hostfs Path: [/]
2022-09-15T12:34:25.257Z ERROR instance/beat.go:1015 Exiting: Failed to create Beat meta file: open /usr/share/heartbeat/data/meta.json.new: permission denied
Exiting: Failed to create Beat meta file: open /usr/share/heartbeat/data/meta.json.new: permission denied

The Azure CLI command customer was using:

$ az container create -g <resource-group-name> --name <container-name> \
  --image docker.elastic.co/beats/heartbeat:7.13.0 --cpu 0.5 \
  --memory 1 --secrets heartbeat.yml="`cat heartbeat.yml`" --secrets-mount-path /mnt/secrets \
  --command-line '/bin/bash -c "heartbeat -environment container --strict.perms=false --path.config /mnt/secrets"'

Customer has not introduced any changes on their side.

We have reproduced the scenario and noticed heartbeat user does not have root group membership like in our original image:

Azure container service:
bash-4.2$ id -a
uid=1000(heartbeat) gid=1000(heartbeat) groups=1000(heartbeat)

Docker engine:
bash-4.2$ id -a
uid=1000(heartbeat) gid=1000(heartbeat) groups=1000(heartbeat),0(root) <--- HERE

That's despite heartbeat being member of this group as per /etc/group file (checked in a container running in Azure):

bash-4.2$ grep heartbeat /etc/group
root:x:0:heartbeat
heartbeat:x:1000:

Lack of root group membership explains permission denied error, as /usr/share/heartbeat/data folder has the following permissions:

bash-4.2$ cd /usr/share/heartbeat/
bash-4.2$ ls -al
total 103364
drwxr-xr-x 1 root      root     4096 May 19  2021 .
drwxr-xr-x 1 root      root     4096 May 19  2021 ..
-rw-r--r-- 1 root      root       41 May 19  2021 .build_hash.txt
drwxrwx--- 1 root      root     4096 May 19  2021 .cache
drwxrwx--- 1 root      root     4096 May 19  2021 .config
drwxrwx--- 1 heartbeat root     4096 May 19  2021 .node
drwxrwx--- 1 root      root     4096 May 19  2021 .npm
-rw-r--r-- 1 root      root    13675 May 19  2021 LICENSE.txt
-rw-r--r-- 1 root      root  8576604 May 19  2021 NOTICE.txt
-rw-r--r-- 1 root      root      848 May 19  2021 README.md
drwxrwxr-x 1 root      root     4096 Sep 30 07:49 data <--- HERE
-rw-r--r-- 1 root      root   249789 May 19  2021 fields.yml
-rwxr-xr-x 1 root      root 96862864 May 19  2021 heartbeat
-rw-r--r-- 1 root      root    69375 May 19  2021 heartbeat.reference.yml
-rw-r--r-- 1 root      root      833 May 19  2021 heartbeat.yml
drwxr-xr-x 2 root      root     4096 May 19  2021 kibana
drwxrwxr-x 2 root      root     4096 May 19  2021 logs
drwxr-xr-x 2 root      root     4096 May 19  2021 monitors.d
drwxrwx--- 2 root      root     4096 May 19  2021 suites
bash-4.2$

We rely on our non-root user belonging to root group to access required directories and it seems ACI is now enforcing a primary group for the default user, stripping root. Has this changed on the container runtime recently?

[elasticmachine]

Pinging @elastic/uptime (Team:Uptime)

[emilioalvap]

To provide a summary of the investigation from Microsoft ACI's team:

• We have confirmed it's a design issue in their container runtime. As it is now, no beats will run out of the box.
• There's no documentation available at the moment that explains this feature or recommended ways to develop containers for the platform.
• Will take feedback into account for future improvements.

Closing this issue as a known limitation.