puppet-zookeeper
puppet-zookeeper copied to clipboard
Default Settings Cause Unwanted ZkServer Process
I'm using this module with the default settings to start a cluster:
class { 'zookeeper':
service_provider => "systemd",
manage_service_file => false,
servers => {
1 => 'ip-10-0-8-10',
2 => 'ip-10-0-16-10',
3 => 'ip-10-0-24-10',
},
}
However, zk fails to come up in quorum mode, instead it comes up in standalone. The root cause of this is something blocking on :2181, and zk complains accordingly:
2017-04-15 06:37:57,711 [myid:1] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
2017-04-15 06:37:57,712 [myid:1] - ERROR [main:QuorumPeerMain@89] - Unexpected exception, exiting abnormally
java.net.BindException: Address already in use
My zoo.cfg:
# .. cut ..
clientPort=2181
server.1=ip-10-0-8-10:2888:3888
server.2=ip-10-0-16-10:2888:3888
server.3=ip-10-0-24-10:2888:3888
Looking at the process table, I can see something is blocking on 2181, and it's a java process calling zkServer. I don't think this is the intended behavior, and I'm not sure what is starting it. This is a brand new host, with nothing on it - I'm only applying this module to it so I assume something is happening with how the init script is built, but that's an arm chair guess.
This only happens when I first run the module. Once I kill the lone zk process and execute systemctl restart zookeeper things come back up as I'd expect.
This is running on Ubuntu 16.04 on AWS.
Any help on this greatly appreciated. Thanks in advance.
Change your manage_service_file to true and it should start up as expected.
What is the output of dpkg --get-selections | grep -v deinstall | grep zookeeper?
It looks like in Ubuntu 16.04 the "service package" zookeeperd contains "old" sysvinit script. Which would mean that ZooKeeper would be started via /etc/init.d/zookeeper file. systemctl status zookeeper should display which script is loaded. If you install a systemd file I'm not sure which one would take precedence. Using service_provider => "systemd" doesn't have any effect as init file is called anyway.
Does it behave differently when with init provider?
service_provider => "init"
$dpkg --get-selections | grep -v deinstall | grep zookeeper
zookeeper install
zookeeperd install
Setting service_provider to init does not seem to help, and neither did setting manage_service_file to true as far as I can tell. On first run of puppet on Ubuntu 16.04, I still get java.net.BindException.
System is a clean Ubuntu 16.04 install.
zookeeper process gets started automatically when zookeeperd package is installed. So guess problem is related to that?