teardown insufficient for Ipmi.Test Ipmi User.Verify IPMI Root User Password Change

[generatz]

I'm often getting fails of this test when I run it back-to-back in quick succession on a development OBMC image. When it fails, the teardown also fails. The teardown runs 'Set Default Password For IPMI Root User' and that fails; the teardown fail message from the log file is as follows:

Keyword 'Set Default Password For IPMI Root User' failed after retrying for 15 seconds. The last error was: Several failures occurred:

1) Error: Unable to establish IPMI v2 / RMCP+ session: 1 != 0

2) 'Error: Unable to establish IPMI v2 / RMCP+ session' does not contain 'Set User Password command successful'

3) Keyword 'Run External IPMI Standard Command' failed after retrying for 15 seconds. The last error was: Error: Unable to establish IPMI v2 / RMCP+ session: 1 != 0

4) TypeError: argument of type 'NoneType' is not iterable

I suggest this test be rewritten to avoid using IPMI in the teardown recovery along these lines:

• Modify the Setup to generate a redfish session token, then the setup attempts to set the original root password (using the new redfish session token to be certain it works) and verifies that the password change succeeded.
• Modify the Teardown so that, as needed, if the default password no longer works, then Teardown restores the original root password using redfish (with the session token created earlier) (I assume, but do not know for certain, that the session token should still be good even if the password is modified.)

[generatz]

There are two interconnected problems within the test case [Teardown]:

1. It does not pause for 5 seconds after Set Default Password For IPMI Root User as it does within the testcase itself.
2. It does not verify that the original password has been restored.

Furthermore, if the original password cannot be restored, the teardown should probably execute a 'Fatal Error'.

Below is a patch that addresses those 3 things. With the patch, one of our development BMCs goes from always failing when the testcase is run repeatedly to always passing. Similarly, for a different manufacturer (with different FW) - the test case was failing often but now is always passing.

I recognize that a really complete error recovery, perhaps using IPMI with redfish backup, is a significant undertaking. Consequently, I'd recommend putting this fix in place now.

$ git diff ipmi/test_ipmi_user.robot
diff --git a/ipmi/test_ipmi_user.robot b/ipmi/test_ipmi_user.robot                                                                                                                                             
index a9d1857d..99b851e0 100644                                                                                                                                                                                
--- a/ipmi/test_ipmi_user.robot                                                                                                                                                                                
+++ b/ipmi/test_ipmi_user.robot                                                                                                                                                                                
@@ -343,9 +343,7 @@ Verify IPMI Root User Password Change                                                                                                                                                      
     [Documentation]  Change IPMI root user password and verify that                                                                                                                                           
     ...  root user is able to run IPMI command.                                                                                                                                                               
     [Tags]  Verify_IPMI_Root_User_Password_Change                                                                                                                                                             
-    [Teardown]  Run Keywords  FFDC On Test Case Fail  AND                                                                                                                                                     
-    ...  Wait Until Keyword Succeeds  15 sec  5 sec                                                                                                                                                           
-    ...  Set Default Password For IPMI Root User                                                                                                                                                              
+    [Teardown]  Verify Root User PW Change Teardown                                                                                                                                                           
                                                                                                                                                                                                               
     # User input password should be minimum 8 characters long.                                                                                                                                                
     Valid Length  OPENBMC_PASSWORD  min_length=8                                                                                                                                                              
@@ -513,6 +511,19 @@ Modify IPMI User                                                                                                                                                                          
                                                                                                                                                                                                               
 *** Keywords ***                                                                                                                                                                                              
                                                                                                                                                                                                               
+Verify Root User PW Change Teardown                                                                                                                                                                           
+    [Documentation]  Teardown for Verify IPMI Root User Password Change                                                                                                                                       
+                                                                                                                                                                                                              
+    Run Keyword And Ignore Error   FFDC On Test Case Fail                                                                                                                                                     
+    Run Keyword And Ignore Error   Wait Until Keyword Succeeds  15 sec  5 sec                                                                                                                                 
+    ...  Set Default Password For IPMI Root User                                                                                                                                                              
+    Sleep  5s                                                                                                                                                                                                 
+    ${v_status}  ${output}=  Run Keyword And Ignore Error                                                                                                                                                     
+    ...  Wait Until Keyword Succeeds  15 sec  5 sec
+    ...  Verify IPMI Username And Password  root  ${OPENBMC_PASSWORD}
+    Run Keyword If  '${v_status}' == 'FAIL'
+    ...  Fatal Error  msg= Original password for root was not fully restored.
+
 Create Random IPMI User
     [Documentation]  Create IPMI user with random username and userid and return those fields.