Switch tests to use pytest

[kloczek]

Looks like pytest is not able to find any units:

+ /usr/bin/pytest -ra
=========================================================================== test session starts ============================================================================
platform linux -- Python 3.8.12, pytest-6.2.5, py-1.10.0, pluggy-0.13.1
rootdir: /home/tkloczko/rpmbuild/BUILD/pwntools-4.7.0
plugins: shutil-1.7.0, virtualenv-1.7.0, mock-3.6.1, cov-2.12.1, anyio-3.3.4, forked-1.3.0, xdist-2.3.0, flaky-3.7.0, tornasync-0.6.0.post2, console-scripts-1.2.0, trio-0.7.0, timeout-2.0.1, asyncio-0.16.0, hypothesis-6.27.0
collected 0 items

========================================================================== no tests ran in 6.29s ===========================================================================

As well setuptools based test suite i sfailing

+ /usr/bin/python3 setup.py test
Failed to convert README.md through pandoc, proceeding anyway
Traceback (most recent call last):
  File "setup.py", line 83, in <module>
    long_description = subprocess.check_output(['pandoc', 'README.md', '--to=rst'], universal_newlines=True)
  File "/usr/lib64/python3.8/subprocess.py", line 415, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/usr/lib64/python3.8/subprocess.py", line 493, in run
    with Popen(*popenargs, **kwargs) as process:
  File "/usr/lib64/python3.8/subprocess.py", line 858, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/usr/lib64/python3.8/subprocess.py", line 1704, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: 'pandoc'
running test
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
/usr/lib/python3.8/site-packages/pkg_resources/__init__.py:116: PkgResourcesDeprecationWarning: 1.16.0-unknown is an invalid version and will not be supported in a future release
  warnings.warn(
/usr/lib/python3.8/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
  warnings.warn(
running egg_info
writing pwntools.egg-info/PKG-INFO
writing dependency_links to pwntools.egg-info/dependency_links.txt
writing entry points to pwntools.egg-info/entry_points.txt
writing requirements to pwntools.egg-info/requires.txt
writing top-level names to pwntools.egg-info/top_level.txt
listing git files failed - pretending there aren't any
reading manifest file 'pwntools.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
warning: no files found matching '*.sh'
warning: no files found matching '*.rst' under directory 'pwnlib'
warning: no files found matching '*.md' under directory 'pwnlib'
warning: no files found matching '*.asm' under directory 'pwn'
warning: no files found matching '*.rst' under directory 'pwn'
warning: no files found matching '*.md' under directory 'pwn'
warning: no files found matching '*.txt' under directory 'pwn'
warning: no files found matching '*.sh' under directory 'pwn'
warning: manifest_maker: MANIFEST.in, line 7: 'recursive-exclude' expects <dir> <pattern1> <pattern2> ...

adding license file 'LICENSE-pwntools.txt'
writing manifest file 'pwntools.egg-info/SOURCES.txt'
running build_ext
Traceback (most recent call last):
  File "setup.py", line 88, in <module>
    setup(
  File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 153, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib64/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib64/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib64/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/site-packages/setuptools/command/test.py", line 223, in run
    self.run_tests()
  File "/usr/lib/python3.8/site-packages/setuptools/command/test.py", line 226, in run_tests
    test = unittest.main(
  File "/usr/lib64/python3.8/unittest/main.py", line 100, in __init__
    self.parseArgs(argv)
  File "/usr/lib64/python3.8/unittest/main.py", line 124, in parseArgs
    self._do_discovery(argv[2:])
  File "/usr/lib64/python3.8/unittest/main.py", line 244, in _do_discovery
    self.createTests(from_discovery=True, Loader=Loader)
  File "/usr/lib64/python3.8/unittest/main.py", line 154, in createTests
    self.test = loader.discover(self.start, self.pattern, self.top)
  File "/usr/lib64/python3.8/unittest/loader.py", line 349, in discover
    tests = list(self._find_tests(start_dir, pattern))
  File "/usr/lib64/python3.8/unittest/loader.py", line 405, in _find_tests
    tests, should_recurse = self._find_test_path(
  File "/usr/lib64/python3.8/unittest/loader.py", line 483, in _find_test_path
    tests = self.loadTestsFromModule(package, pattern=pattern)
  File "/usr/lib/python3.8/site-packages/setuptools/command/test.py", line 56, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib64/python3.8/unittest/loader.py", line 191, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/usr/lib/python3.8/site-packages/setuptools/command/test.py", line 56, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/usr/lib64/python3.8/unittest/loader.py", line 205, in loadTestsFromName
    test = obj()
TypeError: adb() missing 1 required positional argument: 'argv'

[gogo2464]

According to your error log I think you may need to install pandoc (FileNotFoundError: [Errno 2] No such file or directory: 'pandoc')

Then, try to install pandoc first.

In windows : choco install pandoc.

In linux: virtualenv -p python3 venv3 && source venv3/bin/activate && pip install pandoc;

[kloczek]

And here could be some "issue" .. Looks like https://github.com/boisgera/pandoc is not maintained since 2017 and it requitres pandoc which is written in haskel (quite complicated dependencies). Looks like pandoc is used only to process README.md -> rst. IIRC sphinx can process natively .md and/or with some sphinx extension. Really I think using only sphinx extensions or convertion in repo README.md to rst file could simplify build dependencies.

[gogo2464]

For the moment, does my command fix your issue on your computer?

[kloczek]

For the moment, does my command fix your issue on your computer?

Nope /usr/bin/python3 setup.py test still is failing the same way. Other thing is that using setup.py is already deprecated and probably the best way would be migrate test suite to pytest. Talking about deprecations .. looks like pwntools depends on colored-traceback. From pwntools-4.7.1.dist-info/METADATA:

Requires-Dist: colored-traceback

That module is not maintained since 2017 https://github.com/staticshock/colored-traceback.py/

[Arusekk]

If you want to test pwntools, see TESTING.md or .github/workflows/ci.yml. A part of the test suite requires setting up an SSH server anyway, many depend on external network services, and python setup.py test was never intended to work. There are no “unit” tests, only the doctests in the source files showcasing the library usage.

[kloczek]

As I wrote: soon python setup.py test support will be completly removed from setuptools so ..

pytest allows you to start services as part of the test suite (for example ssh server on high port) and kill it at the end of the testing procedure.

[Arusekk]

You still miss my point: our test entry point is $ sphinx -b doctest docs/source docs/build/doctest, not $ ./setup.py test, NEITHER $ pytest. And will likely not be in the forseeable future, because our testing attitude is currently not unit tests (python files with implementation of test cases), but doctests (snippets in function docstrings demonstrating usage of said functions). This is all written in our TESTING.md or if it is not, it is used like this in the CI pipeline description and visible easily from almost every piece of the source code.

[kloczek]

Upu can put that as part of the test suite in pytest.ini. Advantage of that is with generally hanging any tests on pytest tree allows alter testing without changing single line in tested code or test suite definition. It is possible to do that by install pytest extension. For example:

• do you want to perform additional code coverage tests? just install pytest-cov in build env and execute pytest
• do you want to perform some code scanning to find some not-so-well-written-part-of-the-code? Just install pytest-flake8 or pytest-black and execute pytest.

There are tons of pytest extensions which may alter testing that way. This is why pytest is so popular. Just one stat from my rpm spec files:

[tkloczko@devel-g2v SPECS]$ grep ^%pytest python-* | wc -l; ls -1 python-* | wc -l
802
904

Rest of those 102 packages are not using on build procedures pytest because they don't have any test suite. Handfull are still using python setup.py test.

Really I would encorage you to have closer look on pytest because it will be one of the best investment if you are python code developer.