pyslurm icon indicating copy to clipboard operation
pyslurm copied to clipboard
verified publisher icon PySlurm

problem with pyslurm installation

Open gongwei-130 opened this issue 1 year ago • 1 comments

Details

pyslurm installation failed with following cmd: cmd: pip install . error log:

Processing /workspace/trainingstar/third_party/pyslurm Preparing metadata (setup.py) ... done Requirement already satisfied: Cython in /usr/local/lib/python3.9/dist-packages (from pyslurm==21.8.4.0) (3.0.11) Building wheels for collected packages: pyslurm Building wheel for pyslurm (setup.py) ... error error: subprocess-exited-with-error

× python setup.py bdist_wheel did not run successfully. │ exit code: 1 ╰─> [94 lines of output] /workspace/trainingstar/third_party/pyslurm/setup.py:42: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead. if LooseVersion(cython_version) < LooseVersion(CYTHON_VERSION_MIN): /usr/local/lib/python3.9/dist-packages/setuptools/dist.py:491: SetuptoolsDeprecationWarning: Invalid dash-separated options !!

          ********************************************************************************
          Usage of dash-separated 'use-bzip2' will not be supported in future
          versions. Please use the underscore name 'use_bzip2' instead.
  
          By 2025-Mar-03, you need to update your project and remove deprecated calls
          or your builds will no longer be supported.
  
          See https://setuptools.pypa.io/en/latest/userguide/declarative_config.html for details.
          ********************************************************************************
  
  !!
    opt = self.warn_dash_deprecation(opt, section)
  /usr/local/lib/python3.9/dist-packages/setuptools/dist.py:333: InformationOnly: Normalizing '21.08.4.0' to '21.8.4.0'
    self.metadata.version = self._normalize_version(self.metadata.version)
  INFO: running bdist_wheel
  INFO: running build
  INFO: running build_py
  INFO: copying pyslurm/__init__.py -> build/lib.linux-x86_64-cpython-39/pyslurm
  INFO: copying pyslurm/__version__.py -> build/lib.linux-x86_64-cpython-39/pyslurm
  INFO: running build_ext
  Traceback (most recent call last):
    File "<string>", line 2, in <module>
    File "<pip-setuptools-caller>", line 34, in <module>
    File "/workspace/trainingstar/third_party/pyslurm/setup.py", line 384, in <module>
      Pyslurm().setup_package()
    File "/workspace/trainingstar/third_party/pyslurm/setup.py", line 346, in setup_package
      setup(
    File "/usr/local/lib/python3.9/dist-packages/setuptools/__init__.py", line 117, in setup
      return distutils.core.setup(**attrs)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/core.py", line 183, in setup
      return run_commands(dist)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/core.py", line 199, in run_commands
      dist.run_commands()
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/dist.py", line 954, in run_commands
      self.run_command(cmd)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/dist.py", line 991, in run_command
      super().run_command(command)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.9/dist-packages/setuptools/command/bdist_wheel.py", line 407, in run
      self.run_command("build")
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/dist.py", line 991, in run_command
      super().run_command(command)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/build.py", line 135, in run
      self.run_command(cmd_name)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/cmd.py", line 316, in run_command
      self.distribution.run_command(command)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/dist.py", line 991, in run_command
      super().run_command(command)
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/dist.py", line 973, in run_command
      cmd_obj.run()
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/build_ext.py", line 359, in run
      self.build_extensions()
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/build_ext.py", line 476, in build_extensions
      self._build_extensions_serial()
    File "/usr/local/lib/python3.9/dist-packages/setuptools/_distutils/command/build_ext.py", line 502, in _build_extensions_serial
      self.build_extension(ext)
    File "/usr/local/lib/python3.9/dist-packages/Cython/Distutils/build_ext.py", line 130, in build_extension
      new_ext = cythonize(
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 1010, in cythonize
      module_list, module_metadata = create_extension_list(
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 859, in create_extension_list
      kwds = deps.distutils_info(file, aliases, base).values
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 707, in distutils_info
      return (self.transitive_merge(filename, self.distutils_info0, DistutilsInfo.merge)
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 716, in transitive_merge
      return self.transitive_merge_helper(
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 728, in transitive_merge_helper
      for next in outgoing(node):
    File "Cython/Utils.py", line 129, in Cython.Utils.cached_method.wrapper
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 629, in cimported_files
      pxd_file = self.find_pxd(module, filename)
    File "Cython/Utils.py", line 129, in Cython.Utils.cached_method.wrapper
    File "/usr/local/lib/python3.9/dist-packages/Cython/Build/Dependencies.py", line 610, in find_pxd
      pxd = self.context.find_pxd_file(relative, source_file_path=filename)
    File "/usr/local/lib/python3.9/dist-packages/Cython/Compiler/Main.py", line 281, in find_pxd_file
      pxd = self.search_include_directories(
    File "/usr/local/lib/python3.9/dist-packages/Cython/Compiler/Main.py", line 309, in search_include_directories
      return search_include_directories(
    File "Cython/Utils.py", line 90, in Cython.Utils.cached_function.wrapper
    File "/usr/local/lib/python3.9/dist-packages/Cython/Compiler/Main.py", line 703, in search_include_directories
      path = os.path.join(dirname, dotted_filename)
    File "/usr/lib/python3.9/posixpath.py", line 76, in join
      a = os.fspath(a)
  TypeError: expected str, bytes or os.PathLike object, not NoneType
  [end of output]

note: This error originates from a subprocess, and is likely not a problem with pip. ERROR: Failed building wheel for pyslurm Running setup.py clean for pyslurm Failed to build pyslurm ERROR: ERROR: Failed to build installable wheels for some pyproject.toml based projects (pyslurm)

  • Slurm Version: 21.08.5
  • Python Version: 3.9.20
  • Cython Version: 3.0.11
  • PySlurm Branch: v21.08.4
  • Linux Distribution: Ubuntu 22.04.3 LTS

Issue

gongwei-130 avatar Oct 21 '24 23:10 gongwei-130

Hi @gongwei-130,

can you please try installation from this branch?: https://github.com/PySlurm/pyslurm/tree/21.8.x And see if this works. The v21.08.4 tag is not the most up-to-date for the 21.08 version (even though the minor version suggests it), so I just deleted that tag.

tazend avatar Oct 22 '24 07:10 tazend

hi @tazend , 21.8.x can run scripts/build.sh successfully, is it supposed to be installed after the script? I can see it installed at /usr/lib/python3.9/site-packages/, but is under pyslurm-21.8.1-py3.9-linux-x86_64.egg folder. And if I do pip list, it cannot find any package start with pyslurm.

root@abe95083-login:/workspace/trainingstar/third_party/pyslurm/dist# ls /usr/lib/python3.9/site-packages easy-install.pth pyslurm-21.8.1-py3.9-linux-x86_64.egg

NFO: creating 'dist/pyslurm-21.8.1-py3.9-linux-x86_64.egg' and adding 'build/bdist.linux-x86_64/egg' to it INFO: removing 'build/bdist.linux-x86_64/egg' (and everything under it) INFO: Processing pyslurm-21.8.1-py3.9-linux-x86_64.egg INFO: removing '/usr/lib/python3.9/site-packages/pyslurm-21.8.1-py3.9-linux-x86_64.egg' (and everything under it) INFO: creating /usr/lib/python3.9/site-packages/pyslurm-21.8.1-py3.9-linux-x86_64.egg INFO: Extracting pyslurm-21.8.1-py3.9-linux-x86_64.egg to /usr/lib/python3.9/site-packages INFO: Adding pyslurm 21.8.1 to easy-install.pth file INFO: Installed /usr/lib/python3.9/site-packages/pyslurm-21.8.1-py3.9-linux-x86_64.egg INFO: Processing dependencies for pyslurm==21.8.1 INFO: Finished processing dependencies for pyslurm==21.8.1 root@abe95083-login:/workspace/trainingstar/third_party/pyslurm#

gongwei-130 avatar Oct 22 '24 17:10 gongwei-130

never mind, "pip install ." works fine.

gongwei-130 avatar Oct 22 '24 18:10 gongwei-130

Hi @gongwei-130,

yeah the build.sh script does a setup.py build and setup.py install - I've also had cases where this didn't work properly, but usually it does. Installing it with pip install . is definitely the recommended way, maybe I'll just remove scripts/build.sh way from the README. Its mainly used during development, since its easier to build in parallel with multiple cores to cut time. I wasn't able to figure out how to instruct pip install to use multiple cores at the time. (if this is even possible)

tazend avatar Oct 23 '24 06:10 tazend