Rework testing under pytest

[t-bltg]

~~- Rework documentation test so that $ pytest catches https://github.com/vstinner/python-ptrace/pull/70/commits/27f14722679fd0f7e01eee11b68e7122fec2a459.~~

• Added strace.py tests for directory tests/crash (must check if they do not break tox tests added by https://github.com/vstinner/python-ptrace/pull/70).

$ pytest
============================= test session starts ==============================
platform linux -- Python 3.9.1, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: $SRC_DIR
collecting ... 
collected 19 items                                                             

tests/test_crash.py .                                                    [  5%]
tests/test_doc.py .                                                      [ 10%]
tests/test_gdb.py .......                                                [ 47%]
tests/test_strace.py ..........                                          [100%]

============================== 19 passed in 8.44s ==============================

[spoutn1k]

The failure comes from the deletion of test_doc.py that is required in tox.ini. Please edit the file to reflect the changes.

Also, on my machine the module is globally installed, and only running pytest like in your example tests the installation rather than the directory. Please be careful of this. tox is still the only documented testing method, consider adding an entry for pytest.

[spoutn1k]

Why rename test_doc.py to check_doc.py ? This change affects 3 files with no other modifications.

[t-bltg]

Thanks, I'm not familiar with tox, more with pytest (I initially rewrote test_doc.py to unittest because the file was picked by pytest's discovery). Let's rename it to check_doc.py ?.

[spoutn1k]

You may not be but is pytest really necessary ? Victor made the project with tox and adding pytest seems redundant. Does pytest have coverage on anything tox does not ?

Edit - my bad, I made the assumption that tox and pytest were the same general kind of programs, while tox can actually manage pytest runs. Thus we should prioritize running with test tox, and adding pytest as a test driver can be useful. tox still has more features as a test manager - namely virtual environments and embarked parallelism.

[t-bltg]

You may not be but is pytest really necessary ?

I have issues running tox locally, whereas the pytest commands runs fine. Tox sets up a virtualenv and install dependencies, before running the test in a virtualenv installation, I guess pytest is more oriented towards development in the current directory.

[...]
----------------------------------------------------------------------
Ran 18 tests in 7.471s

OK
pep8 create: $SRC_DIR/.tox/pep8
pep8 installdeps: flake8
ERROR: invocation failed (exit code 1), logfile: $SRC_DIR/.tox/pep8/log/pep8-1.log
================================== log start ===================================
ERROR: Could not find a version that satisfies the requirement flake8
ERROR: No matching distribution found for flake8

=================================== log end ====================================
ERROR: could not install deps [flake8]; v = InvocationError('$SRC_DIR/.tox/pep8/bin/python -m pip install flake8', 1)
___________________________________ summary ____________________________________
  py3: commands succeeded

[spoutn1k]

I have issues running tox locally, whereas the pytest commands runs fine.

~~Well flake8 is not installed and fails to install, try running python -m pip install flake8 to see the exact error.~~

Interesting, the sub-installation failed, but this is the formatting check. Does tox have access to the internet on this machine ?

[t-bltg]

~Well flake8 is not installed and fails to install, try running python -m pip install flake8 to see the exact error.~

Interesting, the sub-installation failed, but this is the formatting check. Does tox have access to the internet on this machine ?

It's a virtualenv deployement issue with pip, yes flake8 is installed. Am digging into this, but the logs aren't very helpful ...

[spoutn1k]

Worst case scenario, skip the format check using tox -e py3. Let the github workflow handle the format check.

[t-bltg]

Hi @spoutn1k , @vstinner, do you have any objection to merging this PR ?

[spoutn1k]

Could you remove the github workflow about pytest ? You squashed my commit so I cannot remove it cleanly. It is fine if you prefer to use pytest, but tox is still the preferred and feature-complete way of checking the code.

[vstinner]

You can use python3 -c 'import signal; signum=signal.SIGABRT; signal.signal(signum, signal.SIG_DFL); signal.raise_signal(signum)' (or write a script for that) for example to get a program killed by a signal in a reliable way. So you don't need make/gcc ;-) For faulthandler, I expose private functions in a C extensions for that, like faulthandler._sigsegv().

[t-bltg]

Could you remove the github workflow about pytest ? You squashed my commit so I cannot remove it cleanly. It is fine if you prefer to use pytest, but tox is still the preferred and feature-complete way of checking the code.

Sorry about squashing and cleaning up old commits. It's done.

[t-bltg]

You can use python3 -c 'import signal; signum=signal.SIGABRT; signal.signal(signum, signal.SIG_DFL); signal.raise_signal(signum)' (or write a script for that) for example to get a program killed by a signal in a reliable way. So you don't need make/gcc ;-) For faulthandler, I expose private functions in a C extensions for that, like faulthandler._sigsegv().

Ok, let's forget about running a shell script in the python driven tests.