Add documentation of low-level spike internals

[evancox10]

Hi yall, I'm trying to modify spike to directly control the fetched instruction data, starting from the very first instruction execution. This has been a bit difficult due to a lack of documentation of spike's low level implementation and behavior, and I don't think I'm the only one who's had issues (see #45 #60 #105 #131 )

I wanted to give my input on what information would be good to have, and start by describing some of what I've already discovered.

Examples of what would be good to know:

Devices & Memory Map

• What devices are placed onto Spike's memory map/bus, other than the user-specified RAM?
• What is their default location, size, and alignment? Can this location be changed? Can the devices be disabled?
• What is the function of these devices? How do they accomplish this function?

Implementation Details

• What caches are present in spike? What class is each located in? What are they caching? How are they cleared?
• Where does spike start execution? Can this be changed, and if so how?

Here's a start on the documentation. This is what I've discovered of spike's reset/boot behavior. Maybe this can be added to @poweihuang17 's Spike Documentation repo.

---

Spike Initialization, Boot Sequence, and Default Memory Map

All processors start execution at DEFAULT_RSTVEC. This is hardcoded to 0x1000 in encoding.h, present in both fesvr and riscv-isa-sim. The processor's state.pc variable is initialized with this value when the processor_t constructor calls processor_t::reset(), which then calls state.reset():

void state_t::reset() {
  memset(this, 0, sizeof(*this));
  prv = PRV_M;
  pc = DEFAULT_RSTVEC;
  //...
}

(Other events, such as the assertion of ndmreset in the debug module, will also call processor_t::reset().)

Spike places a boot_rom device at the reset location. This boot_rom contains the reset vector code, followed by the binary device tree data.

The reset vector is a small sequence of code, only 8 instructions, that does the following:

• Load a0 with the value of the processor's mhartid CSR
• Load a1 with the address of the binary device tree data
• Load t0 with the value of start_pc, then unconditionally jump to that location

While the user cannot (currently) alter this boot code sequence, they do have control over the value of start_pc. If the user provides an alternate entry point on the command line, then this will be used. Otherwise, start_pc is initialized with the value of the _entry symbol found in the ELF file.

After the reset vector comes the compiled device tree data. This is the result of compiling the generated device tree string, which can be displayed with the --dump-dts option to spike.

The boot ROM is generated and placed on the bus by the function void sim_t::make_dtb(void). This is first called when simulation begins with sim_t::run(), as shown in the following stack trace:

(gdb) bt 
#0  sim_t::make_dtb (this=0x7fffffffb800) at ../riscv/sim.cc:240
#1  0x00002aaaaad11eea in sim_t::reset (this=0x7fffffffb800) at ../riscv/sim.cc:345
#2  0x00002aaaab249571 in htif_t::start	(this=0x7fffffffb800) at ../fesvr/htif.cc:91
#3  0x00002aaaab24a256 in htif_t::run (this=0x7fffffffb800) at ../fesvr/htif.cc:163
#4  0x00002aaaaad10cf3 in sim_t::run (this=0x7fffffffb800) at ../riscv/sim.cc:94
#5  0x0000000000407827 in main (argc=4,	argv=0x7fffffffcc08) at ../spike_main/spike.cc:153

Currently, there are no run-time or compile-time options to modify this boot behavior.

[poweihuang17]

@evancox10 Thanks your help! We definitely could collaborate on this! Hope we could put our documentation into the repo of spike one day! I will change the documentation into CC-NA-4.0 creative license and add all the contributors into the list.

[poweihuang17]

@evancox10 Do you mind sending a pull request to the repo? Or, I could incorporate your passage by myself. Which one do you prefer?

[aswaterman]

Thanks y'all, this effort is much appreciated.

[mainkar-rohit]

Hi,

Why do we really need a 'reset vector' inside of Spike? Why can't Spike directly start from the _start: address from the ELF?

Is it essential to the working of Spike?

[aswaterman]

We try to keep Spike similar to realistic implementations (which don't have any knowledge about the ELF headers).

[mainkar-rohit]

Hi Andrew,

Thanks for your reply. But that means Spike implements its own boot code that is hardcoded in the source code. This would cause it to deviate from an actual hardware implementation that we want to verify using Spike as a reference model.

Isn't it reasonable to have 'start address at reset (currently 0x1000)' and the 'actual boot code' loadable from an external source/file? This would give Spike flexibility to match and verify with any hardware implementation instruction by instruction.

[aswaterman]

It effectively does have that; you can use --pc=0x1234 on the command line. But it's up to you do do the ELF parsing or whatever.

[mainkar-rohit]

Hi Andrew,

In the current Spike code, the pc changes to the --pc value only after executing the 5 ops from the boot code in Spike. There is no way to bypass these ops.

3 0x0000000000001000 (0x00000297) x 5 0x0000000000001000 3 0x0000000000001004 (0x02028593) x11 0x0000000000001020 3 0x0000000000001008 (0xf1402573) x10 0x0000000000000000 3 0x000000000000100c (0x0182b283) x 5 0x0000000080000000 3 0x0000000000001010 (0x00028067) 3 *0x0000000080000000 *(0x00c0006f) 3 0x000000008000000c (0x00000297) x 5 0x000000008000000c

The --pc can only be used to change *0x0000000080000000 *value. Is that the expected intention with --pc?

Regards, Rohit

On Tue, May 29, 2018 at 2:03 PM, Andrew Waterman [email protected] wrote:

It effectively does have that; you can use --pc=0x1234 on the command line. But it's up to you do do the ELF parsing or whatever.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/riscv/riscv-isa-sim/issues/145#issuecomment-392901647, or mute the thread https://github.com/notifications/unsubscribe-auth/AS7LdOh_IExtq5EiDsl4K8-iFwr5FwBaks5t3ZtqgaJpZM4QBYWB .

[mainkar-rohit]

ping. Since this boot ROM is generated at compilation, it reduces flexibility to bring up Spike with a custom boot sequence and verify a hardware implementation.

[aswaterman]

If you make a pull request to optionally disable the boot ROM (e.g. via a --no-boot-rom) switch, we would be willing to review it.

[dskwe]

@aswaterman Hi, I found the entry address which set by command "--pc==address", should be bigger than DRAM_BASE(0x80000000) in riscv/encoding.h. That means if I set as you metioned below, I found a error like this: ERROR: invalid load from debug module: 8 bytes at 0x00000000000105b8 terminate called after throwing an instance of ‘trap_load_access_fault'

It effectively does have that; you can use --pc=0x1234 on the command line. But it's up to you do do the ELF parsing or whatever.

I have tried turning DRAM_BASE to be 0x03000000, it works. But If I turned it down to 0x02000000, it failed with above error too. I can't start a bare metal program which starts from a small PC just like "0x1234".

I also found program in riscv-tests/isa reset an entry pc 0x80000000, and it would get an error if changed entry pc to 0x70000000. I can't found a solution if I don't change DRAM_BASE in source code. And even after I recompiled, it doesn't work at a small entry pc just like '0x1234'.

So would you like to give an example in which command "--pc=0x1234" works? Or is this a bug?

[aswaterman]

I’m not surprised it doesn’t work with very small addresses. There must be a RAM there, and it mustn’t conflict with other devices (debug and boot ROM).

On Mon, Jul 8, 2019 at 10:15 AM dskwe [email protected] wrote:

@aswaterman https://github.com/aswaterman Hi, I found the entry address which set by command "--pc==address", should be bigger than DRAM_BASE(0x80000000) in riscv/encoding.h. That means if I set as you metioned below, I found a error like this: ERROR: invalid load from debug module: 8 bytes at 0x00000000000105b8 terminate called after throwing an instance of ‘trap_load_access_fault'

It effectively does have that; you can use --pc=0x1234 on the command line. But it's up to you do do the ELF parsing or whatever.

I have tried turning DRAM_BASE to be 0x03000000, it works. But If I turned it down to 0x02000000, it failed with above error too. I can't start a bare metal program which starts from a small PC just like "0x1234".

I also found program in riscv-tests/isa reset an entry pc 0x80000000, and it would get an error if changed entry pc to 0x70000000. I can't found a solution if I don't change DRAM_BASE in source code. And even after I recompiled, it doesn't work at a small entry pc just like '0x1234'.

So would you like to give an example in which command "--pc=0x1234" works? Or is this a bug?

— You are receiving this because you were mentioned.

Reply to this email directly, view it on GitHub https://github.com/riscv/riscv-isa-sim/issues/145?email_source=notifications&email_token=AAH3XQRX6NVPN2ME6FTF6FTP6NY3DA5CNFSM4EAFQWA2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODZNYIDY#issuecomment-509314063, or mute the thread https://github.com/notifications/unsubscribe-auth/AAH3XQVBYEVKJGU2KKVEXSLP6NY3DANCNFSM4EAFQWAQ .

[omerguzelelectronicguy]

Hello, Is there any change in this issue? I have similar problems in my verification processes. I spend a lot of time learning to apply and compile from c to the elf file by using different (linker, crt, etc.) files for each different verification process. I understand and agree that keeping Spike similar to realistic implementations is important as you said. However, I thought it is better to have such a feature bcs many people who have verification of a core started to verify their cores starting from machine mode without any additional needs. If there is not any branch or pull request about it, I will work and try to add this to the spike by making a pull request (--no-boot-rom as you said).