Docker warning

[ghost]

Hi!

After last upgrade of the docker on my host I've started to get warnings like this:

{"timestamp":"2016-08-16T14:19:02.921568+0300","level":"info","message":"publishing check result","payload":{"client":"Gustaff_docker","check":{"command":"check-disk-usage.rb -w 85 -c 95","occurrences":1,"refresh":180,"handlers":["default","mailer","slack"],"name":"disk","issued":1471346380,"executed":1471346342,"duration":0.607,"output":"CheckDisk WARNING: /var/lib/docker/devicemapper/mnt/d707666639bd32898947ad3064223466edaed32646d52281adf81d4d0e170a3e Unable to read., /var/lib/docker/containers/c86190db23e5d247f75b0afd2f150f6d6adcb0de2d2eac630632d468b2e02a1a/shm Unable to read., net:[4026532136] Unable to read., /var/lib/docker/devicemapper/mnt/91fa78b6ebc1e759004a41668945049e5574f7c953efa19036b359059a10a9d7 Unable to read., /var/lib/docker/containers/c4099d43e27f2d6dc45496e4ebbd8e18e4b1fb7ce8d8dc03e63a154801340678/shm Unable to read., net:[4026532202] Unable to read., /var/lib/docker/devicemapper/mnt/0bf4bd7774fd7083fdcb6af2c27f1da45ea4d8fe8fa63fe4067730a59704fb96 Unable to read., /var/lib/docker/containers/a6f6e8d1518f130fb143c3f7a3e77729443717c5b1165b5469692a915468bc44/shm Unable to read., net:[4026532265] Unable to read.\n","status":1}}}

[woltage]

@ulrith You can add parameters: /opt/sensu/embedded/bin/check-disk-usage.rb -p /var/lib/docker -x proc

And after that you won't see errors.

[ghost]

@woltage So I need to create a separate disk check for the Docker host? Does not look like great solution... Any other ideas?

[woltage]

@ulrith hmm.. You are monitoring inside the container and it gives this error?

If you are running Sensu at the Docker hosts: In Docker 1.12.0 you can see the Docker mounts which is different behave versus version 1.10.2. If you look the script:

https://github.com/sensu-plugins/sensu-plugins-disk-checks/blob/master/bin/check-disk-usage.rb#L121

You can see that grabs all mount points and tries to check usage from all the mount points. That's why you are getting the errors. My parameters will ignore those "false" mount points and creates the check to valid ones.

I hope this helps.

[ghost]

@woltage Okay I see. I've added parameters you suggested to my common disk check - thanks!

[majormoses]

related: https://github.com/sensu-plugins/sensu-plugins-disk-checks/issues/53

[majormoses]

@ulrith is there anything else we need to do here? I am thinking we should add some documentation as multiple users have reported. Would you be willing to put together a pr for that?

[rponcetn]

Hi guys!, I have the same issue. This is my check:

{
  "checks": {
    "disk": {
      "command": "check-disk-usage.rb -x tmpfs,overlay,nsfs",
      "subscribers": [
        "all"
      ],
      "handlers": [
        "default",
        "opsgenie"
      ],
      "interval": 300,
      "refresh": 1800
    }
  }
}

This is the disk:

/dev/mapper/docker-202:1-394800-XXXXX on /var/lib/docker/devicemapper/mnt/XXXXX type ext4 (rw,relatime,stripe=128,data=ordered)

Error:

CheckDisk WARNING: /var/lib/docker/devicemapper/mnt/XXXX Unable to read.

How can I fix this warning? I need different checks for docker and instance?

[majormoses]

You can try using this: https://github.com/sensu-plugins/sensu-plugins-disk-checks/issues/52#issuecomment-240625827 to not check that.

[rponcetn]

Thanks @majormoses , recently added -x proc too. But the fs in my case is ext4 and I keep getting this warning.

/dev/mapper/docker-202:1-394800-XXXXX on /var/lib/docker/devicemapper/mnt/XXXXX type ext4 (...

[majormoses]

There are a couple of options:

• configure your system to allow the sensu user to see the volume/dir
• ignore the mount point with -i https://github.com/sensu-plugins/sensu-plugins-disk-checks/blob/2.5.0/bin/check-disk-usage.rb#L46-L49
• enhance to allow ignoring dirs as opposed to mount points or mount fstype via a CLI arg.

I would suggest you go with the -i option for now as that requires no code change. I would love to add more options to provide more flexibility, just wanted to get you a path forward for now.

[rponcetn]

Thanks @majormoses !