riscv-fast-interrupt
riscv-fast-interrupt copied to clipboard
riscv
Proposed reformat of cliccfg
From John Hauser, email March 10, 2019: As currently defined, the CLIC has a single configuration byte, 'cliccfg', that determines the splitting of interrupt control settings into levels and priorities. This one byte applies to all of a hart's privilege levels (M, S, and/or U) simultaneously. I strongly believe this is suboptimal. Instead, each privilege level should be given its own separate configuration. As I'm sure you know, it's common for each privilege level to be running software from different authors, and while the different levels will often interact (through environment calls and such), each privilege level should get to choose for itself how many nested interrupt levels it must accomodate on its interrupt stack, and whether to use selective hardware vectoring. Such choices should not be dictated from above, or from below.
Splitting up 'cliccfg' implies also an adjustment to the way the 'nmbits' field operates. There will now be up to three configuration registers, which here I will call 'clicmcfg', 'clicscfg', and 'clicucfg'. The new registers can have these layouts:
clicmcfg
bits field 7:6 reserved (WARL 0) 5 nmbits 4:1 nlbits[3:0] 0 nvbits
clicscfg
bits field 7:6 reserved (WARL 0) 5 nmbits 4:1 nlbits[3:0] 0 nvbits
clicucfg
bits field 7:5 reserved (WARL 0) 4:1 nlbits[3:0] 0 nvbits
Bit 'clicmcfg.nmbits' determines whether any interrupts can be delegated to privilege levels beyond M-mode. If 'clicmcfg.nmbits' = 0, then all interrupts are taken in M-mode, and 'clicscfg' and 'clicucfg' are ignored. Likewise, when S-mode is supported, bit 'clicscfg.nmbits' determines whether interrupts can be delegated beyond S-mode to U-mode. If 'clicmcfg.nmbits' = 1 and 'clicscfg.nmbits' = 0, then no interrupts are delegated beyond S-mode, and 'clicucfg' is again ignored.
For illustration purposes, let's assume our CLIC implements all 8 bits in each 'clicintctl[]' byte. If 'clicmcfg.nmbits' = 0, then the 8 bits are all used for the M-mode interrupt priority, split up by 'clicmcfg' fields 'nlbits' and 'nvbits', same as before. On the other hand, if 'clicmcfg.nmbits' = 1, then bit 7 of each interrupt control byte decides whether an interrupt is taken in M-mode or is delegated to the next lower privilege level.
The same process then happens again recursively for 'clicscfg', except that at the supervisor level only the lower 7 bits of an interrupt control byte are involved.
Adapting a table from the CLIC document:
---------- clicmcfg clicscfg priv-modes .nmbits .nmbits clicintctl[i] Mode taking interrupt
M 0 none xxxxxxxx M, level+priority=xxxxxxxx
M/U 0 none xxxxxxxx M, level+priority=xxxxxxxx
M/U 1 none 0xxxxxxx U, level+priority=xxxxxxx
M/U 1 none 1xxxxxxx M, level+priority=xxxxxxx
M/S/U 0 - xxxxxxxx M, level+priority=xxxxxxxx M/S/U 1 - 1xxxxxxx M, level+priority=xxxxxxx M/S/U 1 0 0xxxxxxx S, level+priority=xxxxxxx M/S/U 1 1 01xxxxxx S, level+priority=xxxxxx M/S/U 1 1 00xxxxxx U, level+priority=xxxxxx
Just as before, each interrupt source gets configured to be destined for a specific privilege level. For a given interrupt, the configuration byte corresponding to its target privilege level, either 'clicmcfg', 'clicscfg', or 'clicucfg', determines how the remaining interrupt control bits will be split into interrupt level, priority, and SHV bit.
Concerning the implementation cost of this, it is important to realize that the selection of the highest-priority enabled interrupt is not affected by this splitting, but rather depends solely on the value of the control byte in 'clicintctl[]', same as before. Only after the highest-priority interrupt has been identified does it need to be selectively split, which can be handled with a few small muxes of the fields of 'clicmcfg', 'clicscfg', and 'clicucfg', once the target privilege level is identified.
The CLIC memory-mapped control aperture for machine level (the "M-mode CLIC region") should contain all three configuration bytes, 'clicmcfg', 'clicscfg', and 'clicucfg'. If there is a corresponding control aperture for supervisor level, only 'clicscfg' and 'clicucfg' are visible there. A user-level control aperture contains only 'clicucfg'.
Whether or not you agree with me that the implementation cost of this change is insignificant, I believe it's a necessity, for the reasons I gave earlier about separation of control.
preferred naming: mcliccfg, scliccfg, ucliccfg. update memory map 4k regions m/s/ucliccfg starting at 0x0, clicintip/ie/attr/ctl starting at 0x1000 for s/u regions as well. (4k apertures, still recommended priv access based on address)
Discussion was that allowing mode-specific difference in functionality at same address would be problematic in general, and even in this specific case. For example, an M-mode routine trying to access S-mode register would have to know to use mstatus.mprv feature to change effective mode on access. For more general case in other specifications, we've adopted policy of not having mode-specific functional differences on allowed accesses to the same address.
During discussion of this issue in the TG meeting, another option was considered where the boundary between levels was a value instead of done by fixed number of bits. For example, current design requires each privilege mode to have the same number of levels, but user might prefer almost all levels to be made available to one mode (e.g., M-mode has 3 levels, and U-mode has 60 levels). By using a boundary between M and <M modes (e.g., M-mode boundary is 60 indicates levels 60,61,62,63 of an example 6-bit interrupt field are handled in M mode, while lower levels are handled in U-mode), and separate thresholds for X and <X modes for other privilege modes, the allocation of levels to modes would be more flexible. The cost is principally in the comparators at the top of the reduction tree (the tree would have same cost).
This same approach cannot easily be used for priority versus level, and moving to non-equal levels per mode complicates splitting the allocation to each mode between level and priority. One option is to only allow each mode to either treat all values as preemptive levels or queued priorities, but not both.
TG meeting discussed this proposal:
Replace nmbits and nlbits fields with:
mcliccfg.mlvl
scliccfg.slvl
xlvl values are encoded 0 ... 255 (only the most-significant CLICINTCTLBITS are writable in the 8-bit field). Values are encoded as for clicintctl (i.e., unimplemented LSBs are hardwired to 1s).
xlvl field values are absolute level numbers (not relative to mode).
mlvl is lowest level that traps into m-mode. If mlvl=0 all interrupts trap into M-mode. If U-mode is not implemented (implying S-mode is also not implemented), then mlvl=0.
If S-mode is implemented, slvl is lowest level that traps into s-mode. mlvl has priority over slvl, so if mlvl<=slvl no interrupts trap into s-mode. If slvl=0, no interrupts trap into u-mode. If U-mode interrupts are not implemented, then slvl=0.
If U-mode interrupts are implemented, then interrupts <slvl trap into U-mode.
Operation:
The interrupt tree determines the highest-numbered pending-and-enabled interrupts, based on value in clicintctl bits treated as an unsigned integer.
If the value is >= mlvl, the trap is taken in M-mode. otherwise if supervisor mode is implemented and the value is >= slvl, the trap is taken in S-mode otherwise if user-mode interrupts are implemented the trap is taken in U-mode
Is this an issue with using xlvl (I'm calling priv_thresh below) to set the privilege of the interrupt?
Since mcliccfg sets the level at which interrupts are m-mode, s-mode attempts to write clicintctl[__i__] above mcliccfg.priv_thresh should fail because we don't want s-mode to be able to change configuration of an interrupt to m-mode. But with attempts at writing clicintctl[__i__], SW can determine the higher privilege priv_thresh value. Is this a problem?
Benefits to proposal to replace nmbits and nlbits fields with mcliccfg.mlvl and scliccfg.slvl:
- with m and 1 other mode supported, there is the possibility of saving1 implemented flop per interrupt (e.g. m has 4 and u has 12. no savings if m has 5 and u has 12. no savings if m has 8 and u has 8.
- with m and 2 other modes supported, save at least 1 implemented flop per interrupt and it is possible to save 2 flops per interrupt if the packing works out right.
- Interrupts are often not evenly distributed between modes. expect that in many implementations, u or s-mode will have significantly more interrupts than m-mode so the possibility of savings on implementations may be a common implementation occurrence. (However, while m-mode interrupts may need to be carefully prioritized, u or s-mode may be able to group many interrupts into the same interrupt levels. Not clear how significant this benefit really is....)
Disadvantages to proposal:
- interrupt levels have to be coordinated/allocated between modes. software from different groups or different companies have to know how many levels are allocated to each mode. If one mode needs to increase the allocation, other SW for other modes may need to be modified and updated.
- Biggest flop savings occurs on designs with many interrupt levels/many interrupts. Those designs probably aren't as sensitive to the flop count as smaller designs.
- Reduces that max number of interrupt levels (e.g. can't have 255 m-mode, 255 s-mode, 255 u-mode levels. limited to m-mode + s-mode +u-mode levels<=255 (because mintthresh/minstatus is 8 bits).
- would this make adding hypervisor support to CLIC more difficult? Seems messy to need to allocate interrupt levels for hypervisor mode when existence of hypervisor should be invisible to lower priv modes. CLIC would probably need a attr.mode bit to hide hypervisor instead of messing with interrupt levels.
The new proposal would have to ensure supervisor cannot reconfigure an interrupt's settings (clicintattr) then change that interrupt's level (clicintctl) above current mlvl such that M-mode receives interrupts with a configuration it was not expecting. M-mode code could be designed to cope with any configuration, though this seems burdensome.
Alternatively, we can prevent S-mode aperture writes to clicintctl from writing >=mlvl. This is not sufficient by itself, since there is state in the CLIC. If mlvl is changed, M-mode code must also scan through all clicintctl level settings and reset to <mlvl.
To make progress and clean up issue, will make change to add scliccfg then create new thread (issue #226) to discuss new proposal to use levels to separate modes.
Handling the separate cliccfg piece of this issue in #293. The possible new cliccfg format is being handled in issue #226.