coreblocks
coreblocks copied to clipboard
Settle-free testing
This PR is (Yet Another) experiment in making tests simpler to write. The testing mechanism introduced here runs settles when needed, so that a test writer never needs to write a Settle
himself. Tools for synchronizing processes are also introduced. The downside? Use of side effects in process code needs to be restricted to provided primitives, because the process runtime restarts processes behind the scenes (effectively, backtracks them) when synchronization fails.
How is it different from the current way of testing?
- Coroutine syntax (
async
/await
) is used instead of generator syntax (yield
). This makes the code cleaner in my opinion. - No uncontrolled side effects in processes. This includes modifying global variables, queues, and others; however, controlled replacements for these are provided. Debug prints work, but they may be written multiple times per cycle because of backtracking.
- The process code is executed completely once per cycle, there is no primitive for waiting a cycle (like
yield
was doing previously). I'd very much like to have this feature, however - because there is no simple way to implement a nondeterminism effect in Python's coroutines - this can't really be done without making the syntax awful. As most of our testing code is basically in method mocks, this isn't a big limitation. - Signal writes are provided in two varieties:
- Instantaneous writes (like blocking assignment in Verilog); written values are visible by other processes in the same cycle. If there are other signals which depend on the written signal in the simulated circuit, changes are automatically propagated (no settles!).
- Final writes (like nonblocking assignment in Verilog); written values are visible in the next cycle.
- Writes are restricted so that only one process can write a given signal in a clock cycle, and it can do it only once. This is to guarantee that inter-process dependencies are correctly resolved.
- Two synchronization primitives are provided: signals and FIFOs.
- Behavior of signals mirror Amaranth signals, but they are handled by the simulation runtime and can contain arbitrary values. They can be used to store state, and also to synchronize processes (as an instantaneously written value is guaranteed to be seen by processes in the same clock cycle).
- FIFOs allow a single push and pop in a given clock cycle. In doing so, they mirror the behavior of in-circuit FIFOs.
-
TestbenchIO
is not used; instead, method calling and mocking are reimplemented.
As you can see in the tests I've rewritten to use this new mechanism, tests look mostly like before - but there are no Settle
s and sched_prio
s. One can use as many FIFOs, method calls etc. in each process as one wants, and as long as there is no dependency cycle, everything should work just fine.
To-do:
- [x] Minor cleanups. The main simulation code still has some copy-pastes and needs to be refactored a little.
- [ ] Performance improvements. The new mechanism has some performance loss compared to the old one. On my computer, the old retirement test took ~0.75 s, the new one took ~0.95s, a 25% loss. It should be possible to reduce it by scheduling processes in such a way that the number of internal
Settle
s and backtracks are minimized. - [ ] Assertions. It turns out failure is a side effect ;) Currently, consistency is enforced by waiting for values to become stable before calling asserts. This is possibly error-prone, as one must remember to insert the wait. Edit: maybe introduce a read-only phase for processes?
- [x] Controlled debug prints, for easier debugging.
- [ ] A clean way of doing
comb
-like signal assignments is needed. - [ ] A cleaner way for keeping the simulation alive until something happens (e.g. all queues are empty) is needed.
- [ ] Code organization. I'm not sold on all these static methods, but I have no better idea right now.
- [ ] Documentation.
What do you think? Can this save us from the test hell?