How to build man pages when the script requires the current package?

[rrthomas]

I am trying to switch from an old setup.py setup to pure pyproject.toml (thanks for making this possible!) for PSPDFUtils.

The commands I'm trying to make man pages for are standard entry points in my package. With setup.py this works fine; with pure pyproject.toml, I get this error:

Failed to build pspdfutils
  error: subprocess-exited-with-error
  
  × Building wheel for pspdfutils (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]
      No `packages` or `py_modules` configuration, performing automatic discovery.
      `flat-layout` detected -- analysing .
      discovered packages -- ['psutils', 'psutils.command']
      running bdist_wheel
      running build
      running build_py
      running build_manpages
      generating epsffit.1
      Traceback (most recent call last):
        File "/home/rrt/Software/psutils/.tox/py310/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 353, in <module>
          main()
        File "/home/rrt/Software/psutils/.tox/py310/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 335, in main
          json_out['return_val'] = hook(**hook_input['kwargs'])
        File "/home/rrt/Software/psutils/.tox/py310/lib/python3.10/site-packages/pip/_vendor/pyproject_hooks/_in_process/_in_process.py", line 251, in build_wheel
          return _build_backend().build_wheel(wheel_directory, config_settings,
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 410, in build_wheel
          return self._build_with_temp_dir(
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 395, in _build_with_temp_dir
          self.run_setup()
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/build_meta.py", line 311, in run_setup
          exec(code, locals())
        File "<string>", line 1, in <module>
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/__init__.py", line 104, in setup
          return distutils.core.setup(**attrs)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 184, in setup
          return run_commands(dist)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/core.py", line 200, in run_commands
          dist.run_commands()
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 969, in run_commands
          self.run_command(cmd)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-nm5m55mz/normal/lib/python3.10/site-packages/wheel/bdist_wheel.py", line 368, in run
          self.run_command("build")
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/command/build.py", line 132, in run
          self.run_command(cmd_name)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/build_manpages/build_manpages.py", line 151, in run
          self.run_command(DEFAULT_CMD_NAME)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/cmd.py", line 316, in run_command
          self.distribution.run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/dist.py", line 967, in run_command
          super().run_command(command)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/setuptools/_distutils/dist.py", line 988, in run_command
          cmd_obj.run()
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/build_manpages/build_manpages.py", line 127, in run
          parser = get_parser(data['import_type'], data['import_from'], data['objname'], data['objtype'], data.get('prog', None))
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/argparse_manpage/tooling.py", line 66, in get_parser
          return get_parser_from_module(import_from, objname, objtype, prog=prog)
        File "/tmp/pip-build-env-nm5m55mz/overlay/lib/python3.10/site-packages/argparse_manpage/tooling.py", line 28, in get_parser_from_module
          mod = importlib.import_module(module)
        File "/usr/lib/python3.10/importlib/__init__.py", line 126, in import_module
          return _bootstrap._gcd_import(name[level:], package, level)
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 992, in _find_and_load_unlocked
        File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
        File "<frozen importlib._bootstrap>", line 1050, in _gcd_import
        File "<frozen importlib._bootstrap>", line 1027, in _find_and_load
        File "<frozen importlib._bootstrap>", line 1004, in _find_and_load_unlocked
      ModuleNotFoundError: No module named 'psutils'

As we can see, it's not finding psutils. I'm not sure why it works with the setup.py setup but not pyproject.toml.

I'm sorry if this isn't an argparse problem, but it seemed to me that it would be a fairly common problem (that is, wanting to use argparse to make man pages for scripts that require the current package), and I can't see anything in the argparse documentation about it.

[rrthomas]

Just to check that it's not the particularity of one project, I tried with https://github.com/rrthomas/rpl and had the same problem. (I appreciate that this is another of my projects, so it could still be something I'm doing wrong!)

[praiskup]

Thank you for the report. Indeed, this seems like something everyone wants to do when using pyproject+argparse-manpage.. but I don't understand why this is happening.

It seems that psutils is not on the PYTHONPATH. Could setup.py implicitly add the current working directory there, but that's not done by pyproject?

How do you execute the build?

[rrthomas]

I use python -m build. My diagnosis of the immediate problem is the same as yours, and like you I speculated that the path was being set up differently in the two cases; I've not yet had time to probe further.

[superm1]

Did you end up coming up with a solution to this?

[praiskup]

If you are open to experiments, there are IMO two options, a) try to change PYTHONPATH (sys.path) so the missing module is found, or b) make sure that the file with argument parser doesn't import anything fancy (ideally move the parser into a separate file).

[rrthomas]

Thanks @praiskup. Indeed, if I run PYTHONPATH=. python -m build then the build works.

However, I can't always intervene that simply. For example, my tox setup fails while trying to install my package, exactly as above.

I tried using

[testenv]
setenv = 
    PYTHONPATH = .

in my tox config (and several other combinations, with passenv and trying some of tox's pre-defined directories, in particular {toxinidir}, which, being the top level of my project, should be equivalent to . when the command is run in that directory), to no avail.

While I could restructure my code to separate out the parsers, it would be a shame (I have a bunch of short command files which are little more than a parser, but that "little" involves importing from project-defined modules). It feels like I should just be able to set a path somehow in the tox configuration, but after reading it and searching more widely, I'm not sure what or how (what I already did seemed like it ought to work!).

I also can't work out how to add debug to argparse_manpage, as when I edit the files under (in my case) .tox/py312/lib/python3.12/site-packages/argparse_manpage, those edits don't seem to apply to the version used by pip for building my package.

I'm sorry, most of my cluelessness here is nothing to do with argparse_manpage!

[rrthomas]

However, it does feel as though the solution to my problem would be of wide interest, and worth documenting in this package's documentation!

[superm1]

Thanks! I experimented with restructuring but I'm getting even weirder problems.

Baseline (just trying to add argparse.manpage and hitting this issue): https://github.com/superm1/amd-debug-tools/actions/runs/14930226964/job/41944297219

Restructured: https://github.com/superm1/amd-debug-tools/actions/runs/14930530604/job/41945349232

I also tried PYTHONPATH=. python -m build, but it hits same issue as baseline above does.