cassandra-chef-cookbook icon indicating copy to clipboard operation
cassandra-chef-cookbook copied to clipboard

Published 20 hours ago • verified publisher icon michaelklishin

Shutdown issue with JMX Auth

Open rs017991 opened this issue 7 years ago • 5 comments

After enabling JMX Auth, the rhel init.d service is no longer able to perform its graceful shutdown commands via nodetool.

It seems like it should be changed to use the credentials defined in node[:cassandra][:jmx]

[xxxxxx@xxxxxxxxxxx ~]$ sudo service cassandra stop

executing nodetool disablethrift .. nodetool: Failed to connect to 'xx.xxx.xxx.xxx:7199' - SecurityException: 'Authentication failed! Credentials required'. failed to execute disablethrift .. executing nodetool disablegossip .. nodetool: Failed to connect to 'xx.xxx.xxx.xxx:7199' - SecurityException: 'Authentication failed! Credentials required'. failed to execute disablegossip .. executing nodetool drain .. nodetool: Failed to connect to 'xx.xxx.xxx.xxx:7199' - SecurityException: 'Authentication failed! Credentials required'. failed to execute drain .. Shutting down cassandra .. [ OK ]

rs017991 avatar May 04 '17 12:05 rs017991

If by "it" in "it should be changed" you mean nodetool, then it makes sense.

michaelklishin avatar May 04 '17 12:05 michaelklishin

Thanks for the quick response, Michael.

I suppose that I assumed 'it' would be the /etc/init.d/cassandra file which would be changed to include the credentials when it invokes nodetool.

Are you suggesting that the cookbook should instead alter $CASSANDRA_HOME/bin/nodetool file to use those credentials by default?

That would certainly be more convenient(even outside of the context of this issue), though that would have the effect of defeating auth for all local usage of nodetool(which others may have a problem with?)

rs017991 avatar May 04 '17 12:05 rs017991

@rs017991 is this issue occurring on Rhel 7 or 6? I have this enabled in my prod cluster and have implemented both local JMX and remote JMX auth in all my environments and I have never run into this issue. Do you know the steps to replicate the problem?

cmlicata avatar May 04 '17 14:05 cmlicata

CentOS and RHEL 7 use systemd, which means many if not all things configured via /etc/init.d/cassandra will no longer have effect.

michaelklishin avatar May 04 '17 15:05 michaelklishin

@cmlicata We're on RHEL 6.

I went ahead and reproduced the issue using this cookbook's test kitchen (just to make sure I'm not crazy).

  1. Change .kitchen.yml according to this diff
  2. In ChefDK: kitchen converge tarball-centos-68 ; kitchen login tarball-centos-68
  3. Once logged in: sudo service cassandra restart

This produces the same output as my original post.

rs017991 avatar May 04 '17 17:05 rs017991