Failed to run setup.py test

[lahwaacz]

Details

• Slurm Version: 19.05.4.1
• Python Version: 3.8.0
• Cython Version: 0.29.14
• PySlurm Branch: master
• Linux Distribution: Arch Linux

Issue

I'm getting this error when I run python setup.py test (after python setup.py build):

running test
/usr/lib/python3.8/site-packages/setuptools/dist.py:473: UserWarning: Normalizing '19.05.0.0' to '19.5.0.0'
  warnings.warn(
WARNING: Testing via this command is deprecated and will be removed in a future version. Users looking for a generic test entry point independent of test runner are encouraged to use tox.
running egg_info
creating pyslurm.egg-info
writing pyslurm.egg-info/PKG-INFO
writing dependency_links to pyslurm.egg-info/dependency_links.txt
writing requirements to pyslurm.egg-info/requires.txt
writing top-level names to pyslurm.egg-info/top_level.txt
writing manifest file 'pyslurm.egg-info/SOURCES.txt'
reading manifest file 'pyslurm.egg-info/SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'pyslurm.egg-info/SOURCES.txt'
running build_ext
skipping 'pyslurm/pyslurm.c' Cython extension (up-to-date)
building 'pyslurm.pyslurm' extension
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O2 -pipe -fno-plt -D_FORTIFY_SOURCE=2 -fPIC -INone -I. -I/usr/include/python3.8 -c pyslurm/pyslurm.c -o build/temp.linux-x86_64-3.8/pyslurm/pyslurm.o
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_7jobstep___get’:
pyslurm/pyslurm.c:47279:35: warning: comparison of integer expressions of different signedness: ‘int’ and ‘uint32_t’ {aka ‘unsigned int’} [-Wsign-compare]
47279 |     for (__pyx_t_8 = 0; __pyx_t_8 < __pyx_t_7; __pyx_t_8+=1) {
      |                                   ^
pyslurm/pyslurm.c: In function ‘__pyx_f_7pyslurm_7pyslurm_8topology___get’:
pyslurm/pyslurm.c:58346:35: warning: comparison of integer expressions of different signedness: ‘int’ and ‘uint32_t’ {aka ‘unsigned int’} [-Wsign-compare]
58346 |     for (__pyx_t_8 = 0; __pyx_t_8 < __pyx_t_7; __pyx_t_8+=1) {
      |                                   ^
Traceback (most recent call last):
  File "setup.py", line 344, in <module>
    Pyslurm().setup_package()
  File "setup.py", line 305, in setup_package
    setup(
  File "/usr/lib/python3.8/site-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python3.8/distutils/core.py", line 148, in setup
    dist.run_commands()
  File "/usr/lib/python3.8/distutils/dist.py", line 966, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/site-packages/setuptools/command/test.py", line 236, in run
    with self.project_on_sys_path():
  File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__
    return next(self.gen)
  File "/usr/lib/python3.8/site-packages/setuptools/command/test.py", line 155, in project_on_sys_path
    self.run_command('build_ext')
  File "/usr/lib/python3.8/distutils/cmd.py", line 313, in run_command
    self.distribution.run_command(command)
  File "/usr/lib/python3.8/distutils/dist.py", line 985, in run_command
    cmd_obj.run()
  File "/usr/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 186, in run
    _build_ext.build_ext.run(self)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 340, in run
    self.build_extensions()
  File "/usr/lib/python3.8/site-packages/Cython/Distutils/old_build_ext.py", line 195, in build_extensions
    _build_ext.build_ext.build_extensions(self)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 449, in build_extensions
    self._build_extensions_serial()
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 474, in _build_extensions_serial
    self.build_extension(ext)
  File "/usr/lib/python3.8/distutils/command/build_ext.py", line 550, in build_extension
    self.compiler.link_shared_object(
  File "/usr/lib/python3.8/distutils/ccompiler.py", line 713, in link_shared_object
    self.link(CCompiler.SHARED_OBJECT, objects,
  File "/usr/lib/python3.8/distutils/unixccompiler.py", line 158, in link
    lib_opts = gen_lib_options(self, library_dirs, runtime_library_dirs,
  File "/usr/lib/python3.8/distutils/ccompiler.py", line 1090, in gen_lib_options
    lib_opts.append(compiler.library_dir_option(dir))
  File "/usr/lib/python3.8/distutils/unixccompiler.py", line 215, in library_dir_option
    return "-L" + dir
TypeError: can only concatenate str (not "NoneType") to str

[giovtorres]

I don't believe the test subcommand to setup.py was ever tested or even works. Are you trying to run the test suite?

[lahwaacz]

Well, it's apparently doing something, so I thought it should work... How do you run the test suite?

[giovtorres]

I'll have to look at how to set that up properly. After building/installing, I cd out of the directory and run nosetests -v pyslurm/tests.

The tests do need some work, as it needs some test fixtures setup before all the tests can be exercised and verified properly. That is currently not complete.

Feel free to contribute and make this process better :smile:

[lahwaacz]

Requiring a running Slurm instance for a test suite is another unexpected thing. And it still leads to a segfault:

Config: Test config().get() return type. ... ok
Config: Test config().key_pairs() function and return type. ... ok
Config: Compare scontrol values to PySlurm values. ... ok
Hostlist: Create new hostlist ... ok
Hostlist: Test create empty hostlist. ... ok
Job: Test job().submit_batch_job(). ... ok
Job: Test job().get() return type. ... ok
Job: Test job().ids() return type. ... ok
Job: Test job count. ... ok
Job: Compare scontrol values to PySlurm values. ... ok
Job: Test job().find_user() (String). ... ok
Job: Test job().find_user() (Integer). ... ok
Job: Test job().slurm_kill_job(). ... ERROR
Jobstep: Test jobstep().get() return type. ... ok
Jobstep: Test jobstep().ids() return type. ... ok
Jobstep: Test jobstep count. ... ok
License: Test licenses().get() return type ... ok
License: Compare scontrol values to PySlurm values. ... ERROR
Misc: Test slurm_reconfigure() return. ... ERROR
Misc: Test slurm_api_version(). ... ok
Misc: Test slurm_load_slurmd_status(). ... /usr/bin/python3: Segmentation fault (core dumped)

[giovtorres]

These were meant to be acceptance tests, not unit tests.

[lahwaacz]

But they shouldn't segfault in any case...

[giovtorres]

True, although I haven't tested PySlurm with Python 3.8 or on Arch Linux. I'll have to see if I can replicate it locally.

[stdweird]

the original error comes from the fact that running the build step requires passing the "build" on the commandline. if that is not done, the slurm include dir is not set and is passed as None to cython, which cython does not handle/like/expect.

we are facing similar issues when adding pyslurm as a depencency to another project. when we run test for that project, the dependency mechanism tries to install pyslurm, but this is done in a sandbox enviromnnment, where sys.argv is stripped/clean, so no easy way to fix this in our case (except with a patch here)

[tazend]

The test-suite for the ongoing rework of the API has been restructured (at the moment of this writing just on the main branch for slurm 23.02). Tests for the new API have been split into two categories: unit and integration tests. You will need pytest installed to run the tests.

For example, running the unit tests can be done like this:

pytest tests/unit

The Unit tests will not require a running slurm instance - they only make sure that the very core functionalities are working, no interaction with slurmctld/slurmdbd required. (Note: one test might fail currently, as there is a problem with the timezone on a test, will need to fix that soon)

Integration tests can be run with:

pytest tests/integration

Integration tests however require a running slurm instance, as they make all the implemented RPCs to the slurmctld/slurmdbd, to see if everything really works in a live scenario. You should however never execute these integration tests on a production cluster, especially if you are designated as an administrator in slurm, since these tests will make changes and potentially require a specific setup - which is why they will be mainly suited to run by the developers on a local test setup or in github actions.

In regards to running the tests via python setup.py test. As the warning says, setup.py test has been deprecated and is not the recommended (nor supported) way to run the tests - always use pytest directly.

As a sidenote: You cannot run pytest tests directly to run unit and integration tests in one command. They must run seperately at the moment due to the way the tests are imported and searched for by pytest