glances icon indicating copy to clipboard operation
glances copied to clipboard
verified publisher icon nicolargo

The log is spammed with appamor denied messages

Open janusn opened this issue 2 years ago • 9 comments

Describe the bug I have recently switch to Journald logging driver. And I noticed the glances container spams the log with appamor denied messages.

To Reproduce Steps to reproduce the behavior:

  1. Change the log driver of docker to Journald with the following lines added to /etc/docker/daemon.json 
{
  "log-driver": "journald"
}
  1. restart dockerd
  2. Start Glances with the following options '-C /etc/glances.conf -w --password -u ha'
  3. Browse the log with the command $ journalctl -n 1000
  4. See error

Expected behavior There should not be errors related to the container glances.

Environement (please complete the following information)

  • Operating System (lsb_release -a or OS name/version): To be completed with result of: lsb_release -a 
$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 23.10
Release:	23.10
Codename:	mantic
  • Glances & psutil versions: To be completed with result of: glances -V 
# glances -V
Glances v3.4.0.3 with PsUtil v5.9.5
Log file: /tmp/glances-root.log
  • How do you install Glances (Pypi package, script, package manager, source): To be completed glances is installed from a docker container on image 'nicolargo/glances:latest-full'
  • Glances test (only available with Glances 3.1.7 or higher): pastebin

Additional context Add any other context about the problem here. You can also pastebin:

  • the Glances configuration file (https://glances.readthedocs.io/en/latest/config.html#location) pastebin
  • the Glances log file (https://glances.readthedocs.io/en/latest/config.html#logging) Please note for the following log, the PID of the python3 running glances is 1744. pastebin

janusn avatar Nov 23 '23 21:11 janusn

Could you please join the log ?

nicolargo avatar Nov 24 '23 07:11 nicolargo

Could you please join the log ?

Sorry, I am not very familiar with the term. What does "join the log" mean?

Btw, I have attached the output of sudo journalctl -b -g "pid=1744" > 1744.log for your reference and

$ ps aux | grep glances
root        1744 27.5  0.3 182748 121136 ?       Ssl  Nov23 345:09 /venv/bin/python3 -m glances -C /etc/glances.conf -w --password -u ha
janus    1782288  0.0  0.0   6848  2304 pts/0    S+   15:58   0:00 grep --color=auto glances

1744.log

janusn avatar Nov 24 '23 15:11 janusn

Workaround: Add the following option to docker run: --security-opt seccomp:unconfined. It should remove the apparmor message.

nicolargo avatar Nov 25 '23 08:11 nicolargo

Under the hood we need to investigate and understand why Glances needs ptrace access on the system (PsUtil ? another lib ?).

The message are also generated with the "minimal" Alpine Docker Image. So need to investigate on the PSUtil side:

make run-docker-alpine-minimal

Logs (/var/log/syslog file):

Nov 25 09:34:48 XPS13-9333 kernel: [ 2223.756408] audit: type=1400 audit(1700901288.199:569): apparmor="DENIED" operation="ptrace" profile="docker-default" pid=52575 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"

When we disable all the plugins but the process one, the log message is also displayed:

docker run --rm -e TZ="" -e GLANCES_OPT="--disable-plugin all --enable-plugin processcount,processlist" -v /run/user/1000/podman/podman.sock:/run/user/1000/podman/podman.sock:ro -v /var/run/docker.sock:/var/run/docker.sock:ro --pid host --network host -it glances:local-alpine-minimal

So the log are generated by the process plugin (so PsUtil).

nicolargo avatar Nov 25 '23 08:11 nicolargo

Workaround: Add the following option to docker run: --security-opt seccomp:unconfined. It should remove the apparmor message.

I have added the option in a compose.yaml but it doesn't help. The warnings are still reported multiple times every 6 seconds.

---
version: "3"
services:
  glances:
    image: nicolargo/glances:latest-full
    container_name: glances
    
    # temporary fix for silence the kernel warnings reported by journalctl
    security_opt:
      - seccomp:unconfined

    deploy:
      resources:
        limits:
          memory: 1G
    pid: host
    network_mode: host
    environment:
      - "GLANCES_OPT=-w --password -u ha"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - ./etc/glances.conf:/etc/glances.conf:ro
    secrets:
      - source: glances_password
        target: /root/.config/glances/ha.pwd
    restart: unless-stopped
secrets:
  glances_password:
    file: ./secrets/ha.pwd

janusn avatar Nov 25 '23 15:11 janusn

Strange, it makes the job on my side...

Try to add another option (traditionally to the option --security-opt seccomp:unconfined): –-cap-add=SYS_PTRACE

If it did not work, replace the previous options with: --privileged (ugly work around because it give all the system privileges to the container). You can also have a look on the following post (https://stackoverflow.com/questions/63162665/docker-compose-order-of-cap-drop-and-cap-add/63219871#63219871), it should be possible to limit the --privileged option.

Keep us inform if you find a better solution, nothing is possible from the Glances code.

nicolargo avatar Nov 26 '23 08:11 nicolargo

I have tried to replicate your configuration. I have tried 3 tags. And they all spam the log.

Here is my compose.yaml.

---
version: "3"
services:
  glances:
    image: nicolargo/glances:latest-full
#    image: nicolargo/glances:alpine-latest
#    image: nicolargo/glances:latest
    container_name: glances-temp
    pid: host
    network_mode: host
    environment:
      "GLANCES_OPT": "--disable-plugin all --enable-plugin processcount,processlist"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    restart: unless-stopped

janusn avatar Nov 26 '23 19:11 janusn

In my original configuration, neither --security-opt seccomp:unconfined nor –-cap-add=SYS_PTRACE worked but --privileged did the trick.

I tried all 3 options above again with "--disable-plugin all --enable-plugin processcount,processlist" arguments to test on 3 tags

  • latest-full
  • alpine-latest
  • latest

They all crash repeatedly with the following messages logged. (Please note the log messages are listed reverse chronically.)

Nov 26 20:23:31 nuc13 systemd[1]: docker-286f45f2dcf75ddd985016d4140d5ef3055d72d04d2db907bd07c24c8a71132f.scope: Deactivated successfully.
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]: _curses.error: setupterm: could not find terminal
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     setupterm(term=_os.environ.get("TERM", "unknown"),
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/usr/lib/python3.11/curses/__init__.py", line 29, in initscr
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:                   ^^^^^^^^^^^^^^^^
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     self.screen = curses.initscr()
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/outputs/glances_curses.py", line 143, in __init__
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     self.screen = GlancesCursesStandalone(config=config, args=args)
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/standalone.py", line 106, in __init__
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     mode = GlancesMode(config=config, args=args)
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/__init__.py", line 108, in start
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     start(config=core.get_config(), args=core.get_args())
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/__init__.py", line 185, in main
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     glances.main()
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/__main__.py", line 18, in <module>
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "<frozen runpy>", line 88, in _run_code
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "<frozen runpy>", line 198, in _run_module_as_main
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]: Traceback (most recent call last):

Regarding the appamor on with "--disable-plugin all --enable-plugin processcount,processlist" arguments, besides crashing, the docker options made no difference.

  • --security-opt seccomp:unconfined ❌
  • –-cap-add=SYS_PTRACE ❌
  • --privileged ✅
Nov 26 20:23:31 nuc13 systemd[1]: docker-286f45f2dcf75ddd985016d4140d5ef3055d72d04d2db907bd07c24c8a71132f.scope: Deactivated successfully.
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]: _curses.error: setupterm: could not find terminal
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     setupterm(term=_os.environ.get("TERM", "unknown"),
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/usr/lib/python3.11/curses/__init__.py", line 29, in initscr
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:                   ^^^^^^^^^^^^^^^^
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     self.screen = curses.initscr()
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/outputs/glances_curses.py", line 143, in __init__
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     self.screen = GlancesCursesStandalone(config=config, args=args)
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/standalone.py", line 106, in __init__
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     mode = GlancesMode(config=config, args=args)
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/__init__.py", line 108, in start
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     start(config=core.get_config(), args=core.get_args())
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/__init__.py", line 185, in main
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:     glances.main()
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "/app/glances/__main__.py", line 18, in <module>
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "<frozen runpy>", line 88, in _run_code
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]:   File "<frozen runpy>", line 198, in _run_module_as_main
Nov 26 20:23:28 nuc13 286f45f2dcf7[929]: Traceback (most recent call last):
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.275:906105): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906104): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906103): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906102): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906101): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906100): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906099): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906098): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906097): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:23:28 nuc13 kernel: audit: type=1400 audit(1701030208.231:906096): apparmor="DENIED" operation="ptrace" class="ptrace" profile="docker-default" pid=1506038 comm="python3" requested_mask="read" denied_mask="read" peer="unconfined"
Nov 26 20:30:49 nuc13 kernel: kauditd_printk_skb: 1493 callbacks suppressed

janusn avatar Nov 26 '23 20:11 janusn

This issue is stale because it has been open for 3 months with no activity.

github-actions[bot] avatar Jul 08 '24 01:07 github-actions[bot]