sysctl setting req'd by redis fails to apply inside a container

[robbkidd]

From @sean-horn (copied from chef/omnibus-supermarket#47):

Attempting to set this in an lxc container results in this

# sysctl -w "vm.overcommit_memory=1"
error: permission denied on key
'vm.overcommit_memory'

Even though

# id && mount | grep proc
uid=0(root) gid=0(root) groups=0(root)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)

# cookbooks/omnibus-supermarket/recipes/redis.rb
# Redis gives you a warning if you don't do this
  sysctl_param 'vm.overcommit_memory' do
    value 1
 +  # ignore if on lxc.
 +  not_if "grep -q lxc /proc/self/cgroup"
  end

Supermarket's Redis will need this set or background saves may fail. Current conclusion is to output a warning in the Chef run log if converging inside an LXC container and attempt the sysctl change without the guard. This will fail the Chef run with a warning in the output advising the user to make the sysctl setting change in the LXC host system.

[robbkidd]

Experimenting with using ohai data to detect being an LXC or Docker guest along the lines of:

if virtualization[:system] =~ %r{(lxc|docker)} && virtualization[:role] = "guest"

Which is based off of the same sort of checking of /proc/self/cgroup checking.

[robbkidd]

In testing, I've confirmed there is a similar issue with the kernel.{shmall,shmmax} changes we make for PostgreSQL as well.

I notice in my tests that the Chef run itself does not seem to fail. The sysctl cookbook happily writes out our desired changes to a file under /etc/sysctl.d/, but they don't take effect because the container doesn't have the permission to change the kernel parameters. And I don't see its execute of sysctl command failing.

root@u1:~# supermarket-ctl reconfigure
...
Recipe: omnibus-supermarket::redis
  * directory[/var/opt/supermarket/redis/etc] action create (up to date)
  * directory[/var/opt/supermarket/redis/run] action create (up to date)
  * directory[/var/opt/supermarket/lib/redis] action create (up to date)
  * directory[/var/log/supermarket/redis] action create (up to date)
  * template[/var/opt/supermarket/redis/etc/redis.conf] action create (up to date)
  * sysctl_param[vm.overcommit_memory] action apply

  * execute[sysctl[vm.overcommit_memory]] action run
    - execute sysctl -w "vm.overcommit_memory=1" # <------ this is green in the Chef run
  * execute[restart_redis_log_service] action nothing (skipped due to action :nothing)
  * template[/var/log/supermarket/redis/config] action create (up to date)
...

# and yet ...
root@u1:~# cat /proc/sys/vm/overcommit_memory
0
root@u1:~# sysctl -w "vm.overcommit_memory=1"
sysctl: permission denied on key 'vm.overcommit_memory'

@sean-horn Do you have any log data from a Chef run from your testing? Does the Chef run fail on attempting the sysctl change?

[robbkidd]

Moving this to our back log. The proper fix is probably to contribute something to the sysctl cookbook that detects whether it is running inside a container and provides a useful error when changes fail.

[legal90]

It also affects Virtuozzo/OpenVZ containers (virtualization[:system] == 'openvz'). Memory overcommit could be managed by certain options, but not "sysctl". So, this issue makes me to comment/delete the resource sysctl_param[vm.overcommit_memory] declaration each time I upgrade the Supermarket instance 😞 .

https://github.com/chef/supermarket/blob/c3aa536/omnibus/cookbooks/omnibus-supermarket/recipes/redis.rb#L46-L48