mkorpela
Pabot doesn't properly split suites by process with "rerunfailedsuites" option
I am currently using the "rerunfailedsuites" RF option to rerun all failed test cases. This option does not seem to work correctly with pabot. I rerun failed suites with the following command:
pabot --pabotlib --ordering execution_order.dat --processes 12 --rerunfailedsuites 'rerun.xml' ... <some other RF options>... Tests
I successfully use pabot with pretty much these same options on the initial run. I only have problems when rerunning failed suites.
What I see happening is this. Let's say I have 2 tests in 2 separate tests suites that failed (using the examples "11 RON Beta Smoke Test" and "01 Unembed All Signers Sent Env"). pabot starts 2 threads, 0 and 1, as follows. Here we can see we have "11 RON Beta Smoke Test" apparently executing on thread 0, while "01 Unembed All Signers Sent Env" is executing on thread 1:
2021-04-17 22:57:19.721904 [PID:17352] [0] [ID:0] EXECUTING Group_Tests.Scenarios.11 RON Beta Smoke Test
2021-04-17 22:57:19.723161 [PID:17353] [1] [ID:1] EXECUTING Tests.Backend Tests.03 Docusign Tests.03 Unembed Signers.01 Unembed All Signers Sent Env
However, when we get to the test output, we can see that in actuality, BOTH of these tests got executed on thread 1:
------------------------------------------------------------------------------
Tests.Backend Tests.03 Docusign Tests.03 Unembed Signers.01 Unembe... | FAIL |
3 critical tests, 2 passed, 1 failed
3 tests total, 2 passed, 1 failed
==============================================================================
Tests.Backend Tests.03 Docusign Tests.03 Unembed Signers | FAIL |
3 critical tests, 2 passed, 1 failed
3 tests total, 2 passed, 1 failed
==============================================================================
Tests.Backend Tests.03 Docusign Tests | FAIL |
3 critical tests, 2 passed, 1 failed
3 tests total, 2 passed, 1 failed
==============================================================================
Tests.Backend Tests | FAIL |
3 critical tests, 2 passed, 1 failed
3 tests total, 2 passed, 1 failed
==============================================================================
Tests.Scenarios
==============================================================================
Tests.Scenarios.11 RON Beta Smoke Test
==============================================================================
not empty
***** RON Beta Scenario *****
Setup and Send Docusign Envelope | PASS |
------------------------------------------------------------------------------
Get Magic Link | PASS |
------------------------------------------------------------------------------
Notary Enters Conference | PASS |
------------------------------------------------------------------------------
Signer Enters Conference | PASS |
------------------------------------------------------------------------------
Initiate RON Beta Signing Ceremony | FAIL |
Keyword 'Select Frame' failed after retrying for 1 minute. The last error was: Element with locator 'c-conference-share-panel__iframe' not found.
------------------------------------------------------------------------------
Sign and Notarize RON Beta Document | FAIL |
Element 'id=disclosureAccepted' not visible after 45 seconds.
------------------------------------------------------------------------------
RON Beta AV File Download Test | FAIL |
Variable '${notary_invitation_id}' not found.
------------------------------------------------------------------------------
RON Beta DS Journal Link Test | FAIL |
Element 'c-notary-recording-download__go-to-journal' not visible after 1 minute.
------------------------------------------------------------------------------
Tests.Scenarios.11 RON Beta Smoke Test | FAIL |
8 critical tests, 4 passed, 4 failed
8 tests total, 4 passed, 4 failed
==============================================================================
Tests.Scenarios | FAIL |
8 critical tests, 4 passed, 4 failed
8 tests total, 4 passed, 4 failed
==============================================================================
Tests | FAIL |
11 critical tests, 6 passed, 5 failed
11 tests total, 6 passed, 5 failed
==============================================================================
Output: /root/cci_results/pabot_results/1/rerun.xml
Now above, "11 RON Beta Smoke Test" has failed because it was conflicting with the one running on thread 0, which eventually passed:
------------------------------------------------------------------------------
Tests.Scenarios.11 RON Beta Smoke Test | PASS |
8 critical tests, 8 passed, 0 failed
8 tests total, 8 passed, 0 failed
==============================================================================
Tests.Scenarios | PASS |
8 critical tests, 8 passed, 0 failed
8 tests total, 8 passed, 0 failed
==============================================================================
Tests | FAIL |
11 critical tests, 9 passed, 2 failed
11 tests total, 9 passed, 2 failed
==============================================================================
Output: /root/cci_results/pabot_results/0/rerun.xml
At no point does the pabot execution log indicate that "11 RON Beta Smoke Test" is running on thread 1, and yet it is, and it fails, which messes up my rerun test results often due to the conflict.
Not sure if this matters, but here is my execution file:
{
--suite Tests.API Tests.API:Create Room Tests
--suite Tests.API Tests.Browser Version Checks
}
{
--suite Tests.GUI Tests.Room Tests
--suite Tests.GUI Tests.Unsupported Browser Tests.01 Test Block IE
--suite Tests.GUI Tests.Contact Support Test
--suite Tests.Mobile Tests.Mobile Upload Android
}
{
--suite Tests.Backend Tests.01 SFTP Tests
--suite Tests.Backend Tests.02 Email Tests
--suite Tests.GUI Tests.Room Dashboard Tests
}
{
--suite Tests.Backend Tests.03 Docusign Tests.01 DS Conditional Fields
--suite Tests.Backend Tests.03 Docusign Tests.02 DS Prefilled Fields
--suite Tests.Backend Tests.03 Unembed Signers.01 Unembed All Signers Sent Env
--suite Tests.GUI Tests.Ratings Tests.Guest Ratings Tests
}
{
--suite Tests.Scenarios.03 Assurant Auto Smoke Test Android
--suite Tests.Scenarios.03 Assurant Auto Smoke Test iOS
--suite Tests.Scenarios.05 Assurant Home Inspection
}
{
--suite Tests.Scenarios.02 JH GUI Smoke Test Signing Ceremony
--suite Tests.Scenarios.09 RON Smoke Test
--suite Tests.GUI Tests.RON.ID and Verification
}
{
--suite Tests.Scenarios.01 Barclays GUI Smoke Test
--suite Tests.Scenarios.11 RON Beta Smoke Test
--suite Tests.GUI Tests.RON Beta
}
{
--suite Tests.GUI Tests.Admin Functions
--suite Tests.GUI Tests.Admin Preferences Tests
--suite Tests.GUI Tests.Login Tests
}
{
--suite Tests.GUI Tests.No AV Tests.01 Screensharing
--suite Tests.GUI Tests.Room Create Pages
--suite Tests.Scenarios.08 State Farm GUI Smoke Test
}
{
--suite Tests.Scenarios.07 Sales Demo Flow
--suite Tests.Scenarios.10 OneOff Document Smoke
}
I am using the following versions: Robotframework 3.2.1 pabot 1.11
I think this may be a bug. To reproduce it, use an output.xml file with 2 failed test suites and rerun with pabot and the ''rerunfailedsuites" option. If you agree this may be an actual bug, I can come up with a simple script to reproduce, just let me know.
Would also appreciate any ideas on how to get around this situation. Thank you!
