[Tracking] Python configuration and environment issues

[ktbarrett]

#3643 and #4065 show issue with our current solution to Python path configuration, which came from #3309. #4066 was a failed experiment to avoid the first of several of the points below.

Issues

• Some virtual environment tools don't set VIRTUAL_ENV
• Ultimately it doesn't really matter if we are in a virtual environment or not when configuring cocotb
• Setting PYTHONHOME to the virtual environment's prefix doesn't work with venv or tools that use it like pipx
  • This is because the pyvenv.cfg they write doesn't contain base_prefix information, or it isn't setup when Python is initialized for some reason, so it results in base_prefix to incorrectly point to the venv rather than the base.
• Setting the program name to the virtual environment's seems to be sufficient for determine prefix and base_prefix and to setup sys.path as a Python interpreter would, but Py_SetProgramName inexplicably doesn't work on Windows.

Proposed Solution

Based on comments in https://bugs.python.org/issue22213, this is a known issue and there is no interest in fixing it directly. The recommended solution there is to use the Python Config initialization sequence. We would set PyConfig.program_name which would work on all platforms to the python binary gotten using cocotb-config --python-bin, which should return the correct venv executable name even when using isolated package installers like pipx.

This solution is Python 3.8+. Users of Python 3.6-3.7 will continue to experience issues.

[cmarqu]

You may have seen this, but https://bugs.python.org/issue22213 is not updated anymore, https://github.com/python/cpython/issues/66409 has the current discussion.

[Applepi]

Is there an actual solution to fix this on Ubuntu 24.04? I see a bunch of issues but no actual solutions to install the package. installing with pip3 and with pipx results in a broken environment and am unable to run the quick start example.

[ktbarrett]

@Applepi Using pip should work on Ubuntu 24.04 if you set up a virtualenv. If it isn't working you may need to do additional work to configure your environment or there may be something else wrong.

[Applepi]

@Applepi Using pip should work on Ubuntu 24.04 if you set up a virtualenv. If it isn't working you may need to do additional work to configure your environment or there may be something else wrong.

Okay setting up a venv directly seemed to resolve the issue about not finding the package in the venv.

Now it seems to be some other issue when I try to run tests on any of the example test programs, for both iverilog and verilator it says WARNING cocotb.regression No tests were discovered

When I try to run pytest directly I get an error that:

Hint: make sure your test modules/packages have valid Python names.
Traceback:
/usr/lib/python3.12/importlib/__init__.py:90: in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
test_matrix_multiplier.py:19: in <module>
    from cocotb_tools.runner import get_runner
E   ModuleNotFoundError: No module named 'cocotb_tools'

I'm not sure if there is something I'm missing here.

Thanks for your help.

EDIT: It was a problem with the master branch not working, used the latest release and the tests run finish with both simulators. I'm not sure why this project would push non-functional code to master instead of having a separate development branch and then pull in the changes for release.

[FFY00]

Setting PYTHONHOME to the virtual environment's prefix doesn't work with venv or tools that use it like pipx

* This is because the pyvenv.cfg they write doesn't contain base_prefix information, or it isn't setup when Python is initialized for some reason, so it results in `base_prefix` to incorrectly point to the venv rather than the base.

This is expected, PYTHONHOME sets the base prefixes.

Setting the program name to the virtual environment's seems to be sufficient for determine prefix and base_prefix and to setup sys.path as a Python interpreter would, but Py_SetProgramName inexplicably doesn't work on Windows.

On <3.14, prefix/exec_prefix are set to the virtual environment path by the site module, which uses sys.executable to search for the pyvenv.cfg file. If Py_SetProgramName does not work on Windows, as a workaround, you could set Py_NoSiteFlag to disable the site import, fix sys.executable, and then run site.main() to finalize the initialization.

[ktbarrett]

Closed by #4293.