pytest
pytest copied to clipboard
pytest-dev
Misleading error message when parameters are missing in the parametrize but presented in the test function
- [x] a detailed description of the bug or problem you are having
- [ ] output of
pip listfrom the virtual environment you are using - [x] pytest and operating system versions
- [x] minimal example if possible
I'm using pytest version 6.2.4. Say we have the following function and a corresponding test function:
# ./add.py
def add(n1, n2):
return n1 + n2
# ./test_add.py
@pytest.mark.parametrize(
"arg1, expected_answer",
[pytest.param(2, 3, 5)]
)
def test_add(arg1, arg2, expected_answer):
assert add(arg1, arg2) == expected_answer
Running pytest ./test_add.py::test_add logs the following error message:
ERROR: not found: ./test_add.py::test_add
(no name './test_add.py::test_add' in any of [<Module test_add.py>])
But actually it fails because arg2 is not presented in the parametrize decorator.
Similar issue when the argument name is wrong.
This is the full output:
============================= test session starts ==============================
collected 0 items / 1 error
==================================== ERRORS ====================================
_________________________ ERROR collecting test_add.py _________________________
test_add.py::test_add: in "parametrize" the number of names (2):
['arg1', 'expected_answer']
must be equal to the number of values (3):
(2, 3, 5)
=========================== short test summary info ============================
ERROR test_add.py
=============================== 1 error in 0.04s ===============================
ERROR: not found: /home/florian/tmp/test_add.py::test_add
(no name '/home/florian/tmp/test_add.py::test_add' in any of [<Module test_add.py>])
seems quite clear from that what the error is, no?
Yes. But if we can improve the error message in the end, the DX would be even better.
Plus sometimes there are a huge chunk of other logs in between, we have to scroll up to see the real cause of the failure.
The reason why I put up the issue is that I was misled by the error message in the end. I spent some time to figure out why pytest didn’t get to pick up my test function. But in the end it’s the typo inside parametrize that caused the error.
How about we report "couldn't collect" instead of "not found"?
Almost as short, but accurate in a wider variety of cases.
How much complexity will it bring to the codebase/logic if we want to explicitly display to user that test_add was found but it failed to run because there are errors in parametrize?
A simple hint will help a lot, e.g. there are errors in the decorator parametrize. Please refer to the detailed error above
If it is too complex, we can go for a simpler approach: use "couldn't collect" instead of "not found"
We already report that explicitly - see "errors" in the output example above - so I think it's just fixing the "short test summary info" to avoid claiming that any failure to collect was because the test was not found.
currently we really cant know (as there wont be a collection node with the name after collection) since parameterization failed
@The-Compiler i wonder if we could do a indirect parametrize with something that auto marks as "will fail"
alternatively we could have exceptions in generate_tests yield some node that always fails/indicates collection errors
I'd be okay with a simple change to the error message, but anything more complex really doesn't seem to be worth the trouble, IMHO. I really don't think the error message is confusing as-is, personally.
I’ll let the core team make the decision then!
Just trying to propose some rooms for improvement I found (might be very personally and subjectively).
yeah, its a unfortunate edge case, the error itself causes the item not to be "find-able"
what could help is taking a look at collection errors in requested paths and adding a note that a collection error was in the related files
however thats also tricky, so I'd put it in at low prio
As per https://github.com/pytest-dev/pytest/pull/10145#issuecomment-1670239803 I think a fix for this makes the message far more confusing for a way more common case. Thus, closing this.