SystemRDL
Migrate tests to use cocotb
Currently there is a significant amount of duplication between CPU interface testbench library components
Specifically:
tb_inst.svtemplates contain a lot of common boilerplate*_driver.svfiles all implement the exact sameassert_readmethods, among other things
For the tb_inst.sv template, see if it makes sense to autogenerate more of it. Or possibly use a hierarchical Jinja template using blocks & inheritance to make it less verbose. Important: Be careful to not sacrifice clarity. This TB infrastructure is already pretty non-traditional and can be confusing to newcomers.
For drivers, migrate the driver API to live inside SV classes so that they can use proper inheritance.
Or a different wild idea: migrate ALL testbench infrastructure to something like cocotb. SystemVerilog is nice, but much of the verilator compatibility issues are due to fancy things being done in the testbench code and not the generated RTL.
I'll put in my vote for cocotb for a couple of reasons.
- I've used it extensively and I've found it makes testbenches easy to write and reuse.
- It's simulator agnostic. Switching to a different simulator generally requires only telling it which simulator to use. For more advanced use cases it may require custom flags for different tools.
- There's an ecosystem of extensions for various bus interfaces already written (AXI-lite, APB, etc.).
- It ties into pytest very well.
- It could be leveraged by the downstream VHDL exporter. Right now that exporter uses the same Systemverilog testbenches and auto-generates a test adapter between SV and VHDL. Testbenches written in python could work for either language.
Yeah the more I read about cocotb, the more I like it. I've been meaning to learn it for a while and this may just be the perfect excuse to dive into it.
@amykyta3 @darsor
I have rewritten virtually all the tests from regblock into cocotb for my project etana. Part of my verification flow is that my tests still work on the output from regblock.
Because it is open, i can use verilator on github and run actions against every checkin, you can see a cocotb run against the output from regblock using verilator in this action:
https://github.com/daxzio/PeakRDL-etana/actions/runs/18811852950
Cocotb has a problem with structs at the top level in verilator, they should appear as flattened signals that cocotb can just attach to, that is not the case right now (not sure if it is a cocotb or verilator issue) to bypass this i wrote a python script that generates a system verilog wrapper file that provides a mapping of flatten signals to struct signals, hopefully if and when the above issue get addressed the wrapper can be dropped out.
Not every test is there, some tests are more python tests of regblock, rather than simulation tests, so I have as of yet not found an comaprison test to them that cocotb makes sense to do (test_user_cpuif)
* It could be leveraged by the downstream VHDL exporter. Right now that exporter uses the same Systemverilog testbenches and auto-generates a test adapter between SV and VHDL. Testbenches written in python could work for either language.
@darsor
I have also started looking at can this be used for vhdl, yes it can, however the current output of registers that are broken up into fields does not match what verilog does, I think it only supplies the register value at the top level, rather than them broken up into fields, or at least that was as far as i got into the probelm. For tests that did not have the issue they passed staright away against the cocotb test written for the verilog, it was beautiful to see.
@davekeeshan
For tests that did not have the issue they passed staright away against the cocotb test written for the verilog, it was beautiful to see.
That's great! I expect there will need to be some changes to the tests to work with the VHDL exporter, but they should be fairly minor and easy to track relative to upstream. Thanks for testing it!
Having GitHub actions run full simulator verification would be a big win.
@davekeeshan I started dabbling in cocotb yesterday and pretty quickly encountered the struct issue as well with Verilator.
From what I gather, this is a limitation in Verilator, not cocotb. Internally, Verilator doesn't actually support structs at all - instead it mangles structs into flattened arrays as you point out. In a closed design this ends up not making any difference, however when manipulating the design via VPI (what cocotb does) the true representation of the structs shows up - as flat vectors.
This is also why Verilator will throw a warning by default if it encounters an unpacked struct: Verilator is warning the user that the semantics of an un-packed struct is no longer being enforced since it is effectively treating it as packed. This is also apparent when viewing waveforms: VCD dumps only show the flattened structs. Fortunately Verilator is capable of re-building this by passing in the --trace-structs flag.
I'll explore a similar workaround, albeit on the Python side. Rather than flattening in Verilog wrapper, I'd rather auto-generate an interface in Python that re-constructs the struct as a cocotb-equivalent object. That way we preserve an identical user-experience on the TB side so that testbenches can ideally be used across all simulator vendors (ok, except maybe icarus due to very limited SV support...)
Possibly related Verilator issues:
- https://github.com/verilator/verilator/issues/4679 (discusses DPI, so not sure if this also applies to VPI)
@amykyta3
I am not sure how one can connect anything into cocotb if the hooks into verilator are broken, I went looking for anything I could find just instating the SV code from regblock, if you do find something please tell me, I'll feel like an idiot for missing it I am sure.
For the work I needed to do, however, the wrapper was the perfect bridge from where you were to where i needed to be, and got me there pretty fast.
As an observation, I in someways think the most productive and most reusable version of this code is something that looks like almost the straight verilog version of code that I have ended up producing and then it is only as a wrapper around that work that you end up adding in the struct at the top-level, effectively, producing your work. In a code reuse world this hits so many more places and tools. Also when one is producing an open source tool, i personally think one has a mandate to make sure my code works on the free tools first, where possible (icarus, verilator, yosys, ghdl), that is my aspriational goal, but I don't expect anyone to share it.
I have been coding for a long long time, but it is only in the last 10 years I have been writing code and trying to bring that same code through multi tools from different vendors, I am still blown away by how, even some top shelf providers don't do something that has been in the LRM going on 15 to 20 years, the streaming operator issue in xcelium, #165, is a classic example of this. For reference I also found this problem in icarus and fixed it there for that. Last year I was working on some ARM projects, and was surprised (or maybe not surprised when you think about it) the code was written very simply, nothing exotic, pretty much verilog without any system verilog enhancements. I am guessing that comes from the same place, confidence it will work in more places.
@amykyta3
Sorry I meant to say this in the previous post (don't try an post comments on a Sunday afternoon with screaming kids running around), I hope you can use some of my cocotb work, i think it is aboyt 80-85% of what you need right now. I have been actively using cocotb for the past 4 years, and I have had to switch simulation providers a few times, it has been a dream just passing in a different SIM=and it just ... works.
This would be an adapter layer in Python that, if Verilator is being used, would transparently map dut.hwif_in.some.struct.member --> dut.hwif_in[SLICE]. If Verilator ends up fixing this in the future, the shim can be removed and we're back to running 'normally'
Still just an idea, but should be workable. I generally want to avoid flattening since that would add another semantic hurdle when writing testcases. Would prefer the structure to remain unchanged, and instead insert a shim to satisfy Verilator. Agree that flattening would also work, but it seems like an anti-pattern to do that for all simulators just to satisfy Verilator's quirk.
Ah man, all simulator work seems to have some anti pattern problems, hacks upon hacks. When you need to get something out the door you do it with the stated goal of "well get back to it" or "next time", but you don't and then it's sitting there waiting for you on the next project, perpetual groundhog day.
My hack is to stay verilog as much as I can, design is usually 10% of my workload, so it can be tedious upfront it pays out on the backend. There are some nice sv features, but never seem to be worth the pain.
@darsor
OK i have incorporated regblock-vhdl into my cocotb testbench for etana.
https://github.com/daxzio/PeakRDL-etana/actions/runs/19045553460/job/54392747879
it mostly passes, except the regblock testing now has the ability external write-buffered register syntax fixed, #167, this bug still exists in regblock-vhdl and has not been updated yet.
I will raise a issue against that project to capture it.
EDIT: You reported that bug, i think
I may be late to the discussion, but I highly recommend creating a Verilog testbench to wrap the DUT and connecting that testbench to Cocotb. The main reason for this approach is performance. Generating clocks in the Verilog testbench is much more efficient than generating clocks in Cocotb since it eliminates the need to switch between the simulator and Python for every clock cycle.
Additionally, you can write reusable tests using PyUVM, an UVM implementation in Python designed to work with Cocotb. You can generate the UVM register model with peakrdl-pyuvm and utilize the BFMs from the cocotbext-axi package.
@RasmusGOlsen
my work to interface back to both regblock and regblock-vhdl is effectively a wrapper, that is one step off a testbench. I had not heard that about tb generated clocks versus python generated ones, it would be fairly trivial to move that over.
My testing is not UVM but is configured to be reusable, but that was as an act of considered design, which is what we did in the olden days. I want to investigate PyUVM all the same, this might be a good example to start with.
I am far too late and am not trying to back-seat engineer, but reading this thread I am left thinking: why not move to UVM instead? The currently supported simulators all enable UVM and (at least in my experience) is far and away the tool of choice for major companies DV efforts. Is it mainly to support FOSS tooling like Verilator? Or are there some other major benefits I am missing?
I have no experience with CocoTB so I am genuinely interested in understanding.
Totally fair question @allRisc! Yeah the intent is to move more towards a FOSS-friendly testing methodology, but there are also several pretty practical reasons.
Verilator would be the long-term goal since that would also enable testcases to actually be run as GitHub actions. For an open-source project, this is extremely valuable since contributors can then easily test their changes without requiring proprietary tools. Sure, it is possible to get the eval version of Questa via Altera (for now), and one can also use the (impressively buggy) Vivado simulator. Despite these being free, they are fragile options for the general population.
Verilator generally has very good support of the synthesizable subset of SystemVerilog. It still lags behind in several testbench features (discussed in #41), some of which I feel are pretty critical for Verilog-based testbench stimulus. Despite the really impressive strides Verilator has been making lately, I'm still wary about the SV testbench support for some of the stimulus patterns I prefer to use in the testcases.
The benefit of cocotb is that this offers a better way out of this bind: Verilator would only be used to interpret the synthesizable RTL (which it does very well already), and stimulus would be handled by cocotb. There is still a limitation in Verilator's VPI interface regarding cocotb's manipulation of structs, but the path for that being fixed seems far shorter. Sure, I could work around it by doing some clever flattening as @davekeeshan suggested, but that seems like somewhat of a step backwards since it creates additional technical debt I'm not keen on incurring (no offense Dave!!). I'm more invested in getting Verilator improved since then we all benefit.
There are also several ride-along benefits from cocotb that @darsor mentioned earlier that I am also interested in.
One (minor) downside of switching to cocotb is that I don't see clear support for using cocotb with Vivado's xsim[1]. Despite xsim having some really interesting quirks/bugs, it is still one of the few half-decent and semi-freely available simulators. Because of this, I don't anticipate making a full switch until I can get proper Verilator-based sims working.
Regarding UVM: I've used it extensively in my past job and it is a really solid framework that scales well, especially when testing complex systems where vertical-reuse is important. However for this project I feel it would be overkill and would bring more complexity without addressing any of the issues described above.
[1] Worth mentioning: there is some interesting discussion about a cocotb+xsim interface here: https://github.com/cocotb/cocotb/issues/2416
@allRisc, while UVM is a good concept, it doesn't have to be the official SystemVerilog implementation. There is also PyUVM, a Python implementation of UVM that runs in cocotb. For those interested in constrained coverage-driven verification, SystemVerilog still holds an advantage due to SystemVerilog Assertions (SVA), coverage, and effective constraint solving. However, I hope that over time, some Python projects will bridge this gap. In other respects, I believe Python has great potential for verification in the future, and it will be interesting to see how its adoption evolves in the coming years. If it is successful, I'm sure that we will also see support for cocotb in Xsim.
@amykyta3
I would like to provide some context. First i understand your desire to limit technical debt, and while I always try to do the same some times the situation align against me. Case in point, with a properly working verilator integration there would be no need for a shim wrapper file. What is available to icarus in cocotb whould be identical to what should be available in verilator, as specified, a struct flattened down to it component signals. However that is not the case today and I wrote a wrapper script (or my AI copilot did) to look at a verilog top level with struct and create a a flattened top level that allows my cocotb testbench to run on the output of peakrdl-etana and peakrd-regblock (and peakrd-regblock-vhdl). I find this use of AI co-pilots perfect it is a nice and east structured problem for them to understand. When will this be fixed on cocotb/verilator ... I am not keen to try and factor that into a schedule espeically any project that needs to tapeout within the next year.
You could wait for the cocotb guys to fix it, but to now I have experence some fustrarion in term of when they get around to fixing things. There are loads of examples, but my current fustration is how they won't release cocotb-bus, which is a module used in loads of interface models, is broken in cocotb-2.0.0, which is now the release, the code itself was actually edited by me, 2 years ago, because no one else would fix it, and they are multiple requests on the github issues pagge to do a release, and it seems to go straight to /dev/null. My fix, for now, it to copy the current head of that repo's file (it is only one file) into projects i am maintaining, use that instead, and then raise an issue in my github project to come back to it later when they eventually get around to this.
So this is an example of why I am forced to use workarounds, the alternative is that the project stops dead. I also have issue wth verilator compiling xilinx unisims libray which is why I got stuck in icarus in the first place, but you know while verilator may be good for big long sims, if you have a small enough block, icarus can be compiled, running and finished before verilator is finished compiling, that is what keeps me there, but this is more a case of having to find workarounds, the openEDA space is coming along, but there are still areas it struggles (and don't get me started on encrypted IP)
My recommendation is to use the shim (you can use my file its there and it passes all your tests) and get the rest of you testing environment up and running, and then when everything else is working, either put in the hard work to work around it or just put the issue into your to remove later. It is more work than one should have to do (but again some of it is already done) so get the project out there and visible and being used by people is the main threshold to cross, a lesson i learnt quiet a few years into my career is that perfect is the enemy of the good
Any way my cocotb testbench it there, it is free for you to use, i will be maintaining verifing against you code to help keep my code healthy so it is not going to go stale any time soon.
On a final note, thank you for releasing this project, I am slowly introducing it in bits in work, they were very wary at first (mostly documentation) but recently i started to show them how to do the apb/register interface, we have a new project with a significant address map change and it is really beginning to show its worth, so we will only be using rdl more from there on.