DetachHead
Plugin error when suites have tests of the same name
I added pytest-robotframework to my project and I got the following message:
pytest_robotframework._internal.errors.InternalError: something went wrong with the pytedt-robotframework plugin. please raise an issue at https://github.com/detachhead/pytest-robotframework with the following information:
this should NEVER happen, PytestCollector failed to filter out Results Table Should Not Be Empty
pytest==8.2.2 pytest_robotframework==4.3.1 robotframework==6.1 running on Python 3.9.21
are you able to provide a minimal reproduction of the issue? ie. what do your tests look like and what was the command you used to run pytest? thanks
I'm busy with one things at work right now, so bear with me. We're trying to implement a solution with this plugin at work, so I'll definitely be coming back here soon. I'll try to give as much detail as I can without disclosing company information.
It turned out that all's fine when I provide a file to run directly rather than running at project root and selecting the tests by suite tag. I'll be experimenting further. @DetachHead, should I close this issue for now or keep it for future reference?
i think keep it open. as the error message says this should never happen so its likely still an issue that i need to fix. if you can provide exact steps to reproduce it that would be great
I'd have to try and repeat it from scratch as I wouldn't want to risk disclosing any company code. I think this might be a style of execution that some might prefer.
My first finding is that I get the error when I select a directory instead of a specific file, but that's not enough to reproduce the error for a dummy example...
It seems that I've reproduced the issue. It's to do with the fact that in our test suite we have .robot suites that contain tests of the SAME name. Even if I use a tag to filter out the correct test case, the runner fails.
The structure is this: example |- one.robot |- two.robot
The command is: pytest -m hello
Both have a test of the same name. One has a tag of "hello". If you log to console, you will actually see the log of the test with the tag, not the other one.
hmm, i don't know why robot even allows that to begin with. it logs a warning at runtime and robocop has a rule against it too.
pytest-robotframework uses the test name as part of the test's unique node id (eg. "path/to/suite.robot::test name"), so multiple tests in the same suite is bound to cause issues. while i can't reproduce the crash, the behavior i'm seeing is only one of the tests being run and the other one silently being skipped
i'm thinking the fix here is to report a better error message when a suite contains multiple tests with the same name, which means you'll have to rename your tests. would this work for you?
The problem is not that there are tests with the same name within the same suite, but in different suites in the same run.
I'm not sure the execution is correct underneath the errors raised. It seemed like it was.
hmm i can't seem to reproduce that. i can only get it to happen if there are multiple tests with the same name in the same suite