CCI-MOC
error logging for IPMI authentication failure
Per Jason the fix should go in the _ipmitool() function in /haas/ext/obm/ipmi.py.
If haas is using incorrect IPMI credentials for a node, the logged error message report this as failure to power_cycle/power_on/whatever instead of just a failure to log in.
(The end user gets a 500 Internal Server Error, which is fine.)
Ideally, it should:
- recognize and log the access denied error
- skip any further attempts to connect and change the node's power state
- currently, if power_cycle fails on the first try, it sends power_on, in case the failure was because the node was already off
- still return the server 500 error to the user, because we don't want to give out any sensitive info
Here's what the user sees:
[kamfonik@haas-master ~]$ haas node_power_cycle cisco-22
Unexpected status code: 500
Response text:
<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 3.2 Final//EN">
<title>500 Internal Server Error</title>
<h1>Internal Server Error</h1>
<p>The server encountered an internal error and was unable to complete your request. Either the server is overloaded or there is an error in the application.</p>
And here's what gets written to /etc/httpd/error_log about that call:
Error: Unable to establish IPMI v2 / RMCP+ session
Error setting Chassis Boot Parameter 0
Error setting Chassis Boot Parameter 4
[Tue May 03 11:23:34.167243 2016] [:error] [pid 20152] INFO:haas.model:Nonzero exit status from ipmitool, args = ['chassis', 'bootdev', 'pxe']
Error: Unable to establish IPMI v2 / RMCP+ session
Unable to set Chassis Power Control to Cycle
[Tue May 03 11:23:34.179822 2016] [:error] [pid 20152] INFO:haas.model:Nonzero exit status from ipmitool, args = ['chassis', 'power', 'cycle']
Error: Unable to establish IPMI v2 / RMCP+ session
Unable to set Chassis Power Control to Up/On
[Tue May 03 11:23:34.191812 2016] [:error] [pid 20152] INFO:haas.model:Nonzero exit status from ipmitool, args = ['chassis', 'power', 'on']
[Tue May 03 11:23:34.192467 2016] [:error] [pid 20152] ERROR:haas.rest:Server-side failure in function node_power_cycle, raised exception: OBMError(u'Could not power cycle node cisco-22',)
I ran the ipmitool commands with a wrong password, wrong user and no user info this is the output:
Error: Unable to establish IPMI v2 / RMCP+ session
Error sending Chassis Status command
Since the feedback is limited, it is difficult for us to programmatically include an error stating that it was actually access denied due to invalid credentials.
Ideally speaking, nodes that are under HaaS controlled should not be given access bypassing HaaS so as to avoid such situations. But I do recognize that ipmi driver of HaaS needs to do more than what it does now (reporting status of the node, fix the console issues etc) therefore I understand that until we fix that there is going to be such problems.
