Report doesn't display all scenarios when running tests with pytest-xdist

[rafaelcs]

I'm submitting a ...

• [x] bug report
• [ ] feature request
• [ ] support request => Please do not submit support request here, see note at the top of this template.

What is the current behavior?

I'm running a test case on two different devices, but when the allure report is generated, the test case is shown only once.

If the current behavior is a bug, please provide the steps to reproduce and if possible a minimal demo of the problem

Command I'm using to run the tests

1. pytest --dist=each -n 2 --alluredir=allure-results tests/login/test_request_code_without_phone_number.py
2. allure generate -c allure-results -o allure-results/allure-results-html
3. allure serve allure-results/allure-results-html or allure open allure-results/allure-results-html

Report generated:

What is the expected behavior?

Since I am running a test case on two different devices, the report should show repeated scenarios, like this pytest-html report:

What is the motivation / use case for changing the behavior?

Please tell us about your environment:

• Allure version: 2.9.0
• allure-pytest: 2.8.0
• allure-python-commons: 2.8.0
• Appium-Python-Client: 0.47
• pytest: 5.1.2
• pytest-forked: 1.0.2
• pytest-xdist: 1.29.0
• selenium: 3.141.0

Other information

Another question related to this: As I am running a test case on two different devices, how can I display in the report the version of the device that the scenario ran?

[rafaelcs]

Any update here?

[rafaelcs]

Update?

[landhu]

Any update here?NO fix??

[misanthropicat]

What's current status of this issue? Is it still no solution to avoid this?

[delatrie]

This happens because historyId of a test result is calculated based on the nodeid of a test, i.e doesn`t depend on environment in which the test was run.

A historyId value is used to track test retries and history. If multiple tests share the same historyId, allure considers the earliest one as a basic test result and the other ones as its retries. You can see the rest of your tests in the Retries tab:

(Here I`ve run the tests with -n 4)

Allure TestOps (and future allure3) ignores historyId from a test result and calculates it itself based on testCaseId and test parameter values. So, to resolve this issue we should:

1. Detect xdist and add worker id as a hidden dynamic parameter.
2. Also add a python version, platform, hostname if --dist=each as excluded dynamic parameters to show them in the report (excluded means that the parameter is not used when calculating historyId).
3. Make historyId dependent on non-excluded dynamic parameter values to also fix allure2 report.

[M2asXe]

Oh my, this bug hasn't been completely fixed after 5 years, but I'm glad to see some progress. When will it be fully resolved? I'm still troubled by this issue while using python-xdist. @delatrie And if you have any workaround methods, please let me know.

[M2asXe]

Update, I found a workaround to solve this issue. According to PR with ID 745, we can call allure.dynamic.parameter('workid', workid) in Setup\Fixture\Hook, which will allow the Allure report to differentiate parameters and display the results of multiple cases. I have already tested it successfully.