tox icon indicating copy to clipboard operation
tox copied to clipboard

Exit code returned to console does not reflect failure status when running multiple environments

Open szabi opened this issue 2 years ago • 3 comments

Issue

When running a single environment, tox correctly reports the exit code back to the calling shell cmd.exe:

> tox run -e py310-fail 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
FAIL
>
> tox run -e py310-ok 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
>

However, when multiple environments are run, independent of individual results, the error code always indicates "OK"

> tox run -e py310-ok,py310-fail 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
>
> tox run -e py310-fail,py310-fail2 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
>

Expected behaviour:

If at least one run fails, that should be reflected in the return code:

> tox run -e py310-ok,py310-fail 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
FAIL
>
> tox run -e py310-fail,py310-fail2 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
FAIL
>

Of course, if all succeed, that should also be correctly transported:

> tox run -e py310-ok,py310-ok2 1>NUL 2>NUL & if ERRORLEVEL 1 echo FAIL
>

Environment

  • OS: Windows 10
  • pip list of the host Python where tox is installed:
Package    Version
---------- -------
pip        23.0.1
setuptools 67.3.3
wheel      0.38.4

(tox is installed via pipx and is not in that environment).

>tox --version
4.4.6 from <USERHOME>\.local\pipx\venvs\tox\lib\site-packages\tox\__init__.py

Minimal example

I made the effort to set up a minimal example. Feel free to peruse it: See this gist

szabi avatar Mar 14 '23 17:03 szabi

PR welcome to fix this 😊

gaborbernat avatar Mar 27 '23 16:03 gaborbernat