agent-js-playwright
agent-js-playwright copied to clipboard
reportportal
The failed tests appear repeated instead of appearing grouped and with the corresponding retries
Hi RP team! I'm using Report Portal with Playwright and I have the following issue: The failed tests appear repeated instead of appearing grouped and with the corresponding retries:
The 'Unique test case id' is the same for both. The test in my console and in the Playwright report appear grouped together.
Versions:
- "@playwright/test": "^1.40.1"
- "@reportportal/agent-js-playwright": "^5.1.6"
Any idea what the problem could be? Thanks in advance.
Hi @lucas-delafuente-deel. We are currently working to resolve this issue. Does this issue occur constantly or occur with some frequency? If you have case that reproduces this issue exactly, could you share that information with us?
Hi @AliakseiLiasnitski, it is weird because I used report portal in the past and it worked fine, I will try with old versions of the report portal agent.
- It's not always with the same tests, it seems to be something random, I couldn't find a pattern (update -I confirm that it always happens with the first test that fails, the following ones that fail are grouped well)
- Example: if I run 60 tests and 4 tests are unstable and have two or one retries, one of those tests appears repeated instead of grouped, from what I saw it seems that it is always the first test that fails and has retries, that one doesn't appear repeated, the rest of flaky or fail test appear grouped with their retries.
First test is not grouped, second test is grouped:
I appreciate the work you do to solve this bug 🚀 , I was wondering if there is any estimate of when the solution will be available?
Thank you so much.
Hello @lucas-delafuente-deel Thank you very much for your observations, I will take into account everything you wrote above.
We currently do not know when this issue will be resolved, but we will keep you updated and write to you as soon as we resolve this issue.
Hi @lucas-delafuente-deel We have improved our approach to retry tests and released these changes. To solve your problem, I would like to ask you to install the latest version of @reportportal/agent-js-playwright (5.1.7) and check if everything works correctly for you. If this error continues to occur, please contact us.
Hi team! thanks for the update 🚀 , I will try with that version!
We are testing @reportportal/agent-js-playwright (5.1.7), so I will send the update in a few days to see if we continue to see the issue.
Hi @lucas-delafuente-deel ! Thanks for testing the latest version. Have you encountered any occurrences of this issue in the meantime?
Hi, sorry for the delay! We still have the same issue and in addition, we are not having the screenshot and the trace in the tests.
Hi @lucas-delafuente-deel ! Sad to hear that. Looks like we have another cause for this issue. Can you please tell us if you are using manual provisioning of testCaseId? We have a hunch that this may be related.
Regarding the missing screenshot and trace. Are they missing for all tests or for some of them? Are there any errors in the console output?
Sorry for the delay. The issue about the screenshot and trace are related to minIO from Report Portal, our devops is investigating that.
I will try with manual provisioning of testCaseId.
Hello @lucas-delafuente-deel ! Hope you are doing well! Do you have any news related to this issue? Does it still exist?
Hi @AmsterGet sorry for the delay, yes, we are still seeing the issue (less frequently than before) , I haven't had time to test the approach with testCaseId yet 🥲
@lucas-delafuente-deel Understood. Anyway, if you can gather more information related to the problem, please share it with us. Ideally, it would be great to have a reproducible example to help us identify the issue.