cocotb icon indicating copy to clipboard operation
cocotb copied to clipboard

pytest-cocotb?

Open oscargus opened this issue 8 months ago • 3 comments

With the new pytest-driven approach it would be nice to be able to pass arguments to pytest. rather than through environment variables. For this a pytest extension would be nice. If nothing else, being able to pass, say, --sim=nvc setting the environment variable SIM to nvc for the current run. This will lead to a more consistent experience IMHO and will make it possible to list arguments as part of pytest --help. Also, some currently long environment variables can be accessed in a shorter way.

To me, it would make sense that this is hosted under the cocotb organization, although one can of course imagine starting somewhere else. I guess one can maybe create this as part of the cocotb package as well (with the benefit of being able to add arguments without have a corresponding environment variable etc). If I understand correctly, cocotb is already registering to pytest?

oscargus avatar Apr 27 '25 09:04 oscargus

This sounds nice, but it should definitely be a separate package. There are many possible uses of pytest with cocotb so anything that prevents other users from using cocotb with pytest in their own way should be developed separately. But if it can be included in the cocotb package in some way which is opt-in, that's also fine.

Also I'm not so sure about how stable the pytest plugin API is, but if it changes a lot, putting that package in the cocotb repo might not be a good idea as cocotb moves a lot slower than might be required to best support such a plugin (but maybe not, I don't know).

If I understand correctly, cocotb is already registering to pytest?

I'm not sure what you mean, so I'm going to guess no.

ktbarrett avatar Apr 27 '25 14:04 ktbarrett

I'm not sure what you mean, so I'm going to guess no.

https://github.com/cocotb/cocotb/blob/0b47978253170dceec0e6ff4e0245daf64b19cb6/src/cocotb/regression.py#L202

sort of. That cocotb already now interacts with pytest.

I think that one can, maybe, do something similar with pytest arguments. But I get the point of having a separate package. On the other hand, passing things with environment variables may be quite tedious and it can be hard to guarantee that they are only set temporarily (see https://stackoverflow.com/questions/2059482/temporarily-modify-the-current-processs-environment for options).

oscargus avatar Apr 28 '25 08:04 oscargus

The assertion writing is a bit different. We aren't registering ourselves into pytest, but registering pytest's assertion rewriting import hook. It works standalone. Same with cocotb supporting pytest.raises, pytest.warns, and in 2.1 pytest.skip and pytest.xfail. cocotb doesn't insert itself into the pytest flow in any way right now.

Based on https://docs.pytest.org/en/stable/how-to/writing_plugins.html we could keep the plugin code in the cocotb repo and require users to load the plugin via conftest.py or by adding -p cocotb_tools.runner to the commandline call or their pytest.ini. That makes it opt-in. We will not add an entry point plugin.

ktbarrett avatar Apr 28 '25 17:04 ktbarrett