Tracking: Support Tock in QEMU and use it for CI

[alistair23]

This is an overall tracking issue for support Tock in QEMU. Then we can use QEMU to test Tock in the CI.

• [x] Convert the current QEMU testing set-up to use a test-runner (or something else more capable then what we have now).
• [x] Add a make qemu target to OpenTitan when OT support is merged into mainline QEMU
  • [x] Add CI testing for OpenTitan when OT support is merged into mainline QEMU
• [ ] Add support for running the ARM port on QEMU
  • [ ] The nucleo_f429zi binary runs on the Netduino 2 Plus QEMU machine
    • [ ] Fix the reset order/flash alias CPU reset bug for v7m CPUs in QEMU
    • [ ] Add support for the SMT32Fxx DMA (required for USART)
    • [ ] Add a make qemu target

[rajivr]

Add support for the SMT32Fxx DMA (required for USART)

Are there plans for opentitan to support DMA peripheral? From what I can see from here, there seems to be no plans to add DMA peripheral.

When I wrote the STM32 DMA for USART, I was very new to Tock, and following the shuhari principle, I pretty much copied what Amit had done with SAM4L.

Right now I am not sure how stable of a dma1::StreamClient or dma::DMAClient trait we would end up having upstream. It might actually make more sense to move to a purely interrupt driven USART for STM32 QEMU.

[alistair23]

For OpenTitan I don't see DMA mentioned in the docs. Someone from the OpenTitan team would need to answer if they are going to add it.

I'm not sure what you mean about a stable dma1::StreamClient or dma::DMAClient.

An interrupt driven USART driver in Tock would also address the issue (then QEMU wouldn't need the DMA device).

[rajivr]

I'm not sure what you mean about a stable dma1::StreamClient or dma::DMAClient.

Unlike traits in kernel/src/hil traits, DMA trait in internal to the chip crate. So, it doesn't get the level of scrutiny that the HIL traits gets, but can have a huge impact on how we support chip peripherals.

[alistair23]

I'm happy to swap the STMF4xx chips to not use DMA for USART. I don't know enough about Tock and ARM/STM to know if that's the right move or not.

I don't have an STM32F4xx board to test, so I won't be able to do it. If someone does the conversion I can probably get it running in QEMU then.

[dpinthinker]

I don't have an STM32F4xx board to test, so I won't be able to do it. If someone does the conversion I can probably get it running in QEMU then.

@alistair23 I can buy a STM32F4xx board for the testing. Which board do you prefer?

[alistair23]

So someone will need to edit the USART code and then test it (if that's the correct solution). You would want to test it on one of the STM32F4xx boards that Tock supports.

[dpinthinker]

So someone will need to edit the USART code and then test it (if that's the correct solution). You would want to test it on one of the STM32F4xx boards that Tock supports.

How about this one?

STM32 Nucleo-64 development board with STM32F446RE MCU

https://github.com/tock/tock/blob/master/boards/nucleo_f446re/README.md https://www.st.com/en/microcontrollers-microprocessors/stm32f446re.html

[alistair23]

I think before you buy one you should check (maybe on Slack?) if it's the correct solution. If you want to buy one to mess around with anyway then go ahead. It looks like there are links with the README on where to buy the boards.

[dpinthinker]

@alistair23 I just got my new board STM32 Nucleo-F446RE. What can I do next?

[alistair23]

Awesome! So the next step is to convert the USART driver to use interrupts instead of DMA. Then test that on the board. Is that something you think you can do?

[dpinthinker]

Awesome! So the next step is to convert the USART driver to use interrupts instead of DMA. Then test that on the board. Is that something you think you can do?

Just found that chip stm32f303xc used interrupts to implement USART driver. https://github.com/tock/tock/blob/master/chips/stm32f303xc/src/usart.rs#L483

Hi @garasubo @alexandruradovici stm32f4xx USART driver is implemented by DMA. stm32f303xc USART driver is implemented by interrupts.

Could you both give some help info about how to swith [stm32f4xx USART driver] from DMA to interrupt?

The reason to do that is we want to enable QEMU support for STM32 board.

[alexandruradovici]

Awesome! So the next step is to convert the USART driver to use interrupts instead of DMA. Then test that on the board. Is that something you think you can do?

Just found that chip stm32f303xc used interrupts to implement USART driver. https://github.com/tock/tock/blob/master/chips/stm32f303xc/src/usart.rs#L483

Hi @garasubo @alexandruradovici stm32f4xx USART driver is implemented by DMA.

I wrote the stm32f3 USART without DMA as I did not have the time to setup the DMA on that chip, but it is something that I want to do.

stm32f303xc USART driver is implemented by interrupts.

Could you both give some help info about how to swith [stm32f4xx USART driver] from DMA to interrupt?

I can try to convert the stm32f4 to non DMA, I think the hardware is similar top the stm32f3. The question is how do you want to do this? As just a feature for testing purposes? It would make a lot of sens to have the DMA when running on the real hardware.

The reason to do that is we want to enable QEMU support for STM32 board.

[dpinthinker]

I can try to convert the stm32f4 to non DMA, I think the hardware is similar top the stm32f3. The question is how do you want to do this? As just a feature for testing purposes? It would make a lot of sens to have the DMA when running on the real hardware.

My original requirement is that our team just have one STM32 board(Nucleo-F446RE) supported by tock os. If STM32 QEMU is supported, it would help a lot.

And then @alistair23 tried to do that, but he found USART not working on QEMU with DMA.

What if USART just switch to interrupt in QEMU and the real hardware keep using DMA? Is it possible?

@alistair23 What is your opinion about this?

[alexandruradovici]

What about having a special stm32f4qemu crate where we swap out the uart with a non-DMA uart?

[dpinthinker]

What about having a special stm32f4qemu crate where we swap out the uart with a non-DMA uart?

I think it is fine to me.

[ppannuto]

What about having a special stm32f4qemu crate where we swap out the uart with a non-DMA uart?

I think I might be slightly opposed to this. Ultimately, what is the point of QEMU support? -- I see it largely as allowing for testing and validation without requiring real hardware. To that end, having "different" boards/chips for emulation and hardware is a step backward.

I would rather the effort go to fixing DMA support in QEMU than trying to create a separate, QEMU-specific board in Tock.

If there is a platform that is only available in emulation, then I think Tock might entertain a specific board for it. I don't think Tock would be inclined to merge a board / chip that is just a workaround for a broken/incomplete emulator.

[alistair23]

Just a quick update. The RISC-V testing has progressed pretty well and we are now running mainline QEMU to test both the HiFive1 and OpenTitan. There is work underway to test userspace apps as well for those platforms.

[mars-low]

Just a quick update. The RISC-V testing has progressed pretty well and we are now running mainline QEMU to test both the HiFive1 and OpenTitan. There is work underway to test userspace apps as well for those platforms.

Has it actually become finalized to test userspace apps in QEMU? Currently I'm trying to run userspace apps on OpenTitan earlgrey-nexysvideo platform as per instructions in directory https://github.com/tock/tock/blob/e02c5629b26c648febaab587fb79c3ddce2d2077/boards/opentitan/README.md?plain=1#L170-L186 and it doesn't work for me.

I'm getting message:

I00001 boot_rom.c:76] Boot ROM initialisation has completed, jump into flash! App flash does not match requested address. Actual:0x20030040, Expected:0x20030000 OpenTitan initialisation complete. Entering main loop

It was after loading libtock-c/examples/c_hello/build/rv32imc/rv32imc.0x20030000.0x10008000.tbf , which was build after adding custom target rv32imc|rv32imc.0x20030000.0x10008000|0x20030000|0x10008000 to https://github.com/tock/libtock-c/blob/c0ec4fba3d9fa150845e810c29f13bcbf80e0266/Configuration.mk#L60-L63. I thought that 0x20030000 should be valid address to use as the fixed start of flash for opentitan platform.

Actually I tested other built binaries as well and it didn't work. Often it was just jumping to main loop without printing any error which probably means it hasn't been detected by kernel:

OpenTitan initialisation complete. Entering main loop

Other tested binaries produced during building include:

rv32imc.0x20030040.0x10008000.tbf rv32imc.0x20030080.0x10005000.tbf rv32imc.0x20030880.0x10008000.tbf rv32imc.0x20032080.0x10008000.tbf rv32imc.0x20034080.0x10008000.tbf rv32imc.0x41000060.0x42008000.tbf

and

rv32imac.0x403B0060.0x3FCC0000.tbf rv32imac.0x20040060.0x80002800.tbf rv32imac.0x40430060.0x80004000.tbf rv32imac.0x40440060.0x80007000.tbf

I've also tried to run app from recent libtock-rs but saw that it is not yet compatible with Tock OS 2.0 and it didn't work as well.

I've seen that kernel itself and unit tests are running on ci-qemu and I can also reproduce it locally. However I'm not able to load userspace apps in QEMU on master branch.

I've been able to run userspace apps using Tock release-1.6 and libtock-c release-1.6 by running command make OPENTITAN_BOOT_ROM=/workspaces/tock/tools/qemu-runner/opentitan-boot-rom.elf APP=/workspaces/libtock-c/examples/c_hello/build/rv32imc/rv32imc.0x20030080.0x10005000.tbf qemu-app

Working target was rv32imc.0x20030080.0x10005000.tbf in that case.

[alistair23]

You need to use the 0x20030080.0x10005000 application on OpenTitan. If it isn't working you can file an issue. One hint to help debug is to enable extra prints in kernel/src/config.rs

[mars-low]

Sorry for the confusion. I've verified my setup by running QEMU emulation in clean environment and got it working with latest Tock kernel and libtock-c library. So it was definitely problem on my side. Nevertheless thanks for hints about debugging and proper application. In the meantime I've found out that there is board-runner for earlgrey_nexysvideo which helps a lot to deduce what tbf binaries to load. I see that on real board more apps can be loaded without problem.

I'm trying to reproduce it with QEMU. Please correct me if I have wrong assumptions about loading apps built from libtock-c repository.

I see that there are four targets for OpenTitan in libtock-c configuration files:

OPENTITAN_TOCK_TARGETS := rv32imc|rv32imc.0x20030080.0x10005000|0x20030080|0x10005000
rv32imc|rv32imc.0x20030880.0x10008000|0x20030880|0x10008000
rv32imc|rv32imc.0x20032080.0x10008000|0x20032080|0x10008000
rv32imc|rv32imc.0x20034080.0x10008000|0x20034080|0x10008000

I suppose that I can load up to four apps for Tock kernel on OpenTitan platform and they are ordered in memory as given by their fixed flash adresses:

0x20030080 0x20030880 0x20032080 0x20034080

However looking at the method earlgrey_nexysvideo_blink_and_c_hello_and_buttons() shows that apps are loaded in slightly different order:

0x20030080.0x10005000 0x20030880.0x10008000 0x20034080.0x10008000

and are concatanated before flashing on board so my previous deduction is probably wrong.

Now I'm trying to load two apps in QEMU. From earlgrey-nexysvideo board directory I'm running following command:

.../../../tools/qemu/build/qemu-system-riscv32 -M opentitan
-kernel /workspaces/tock/target/riscv32imc-unknown-none-elf/release/earlgrey-nexysvideo.elf
-bios /workspaces/tock/tools/qemu-runner/opentitan-boot-rom.elf
-device loader,file=/workspaces/libtock-c/examples/c_hello/build/rv32imc/rv32imc.0x20030080.0x10005000.tbf,addr=0x20030000
-device loader,file=/workspaces/libtock-c/examples/cxx_hello/build/rv32imc/rv32imc.0x20030880.0x10008000.tbf,addr=0x20030800
-nographic -serial mon:stdio

Second application defines its address in flash as 0x20030880 and I load tbf binary at 0x20030800 to take into account header size which is 0x80 (is it true?).

Anyway second app is not loaded with output:

QEMU output for OpenTitan Earlgrey Nexys Video platform

I00000 boot_rom.c:60] Version: opentitan-earlgrey_silver_release_v5-157-ga28c280bb Build Date: 2021-06-30, 02:13:50

I00001 boot_rom.c:76] Boot ROM initialisation has completed, jump into flash! Loading processes from flash=0x20030000-0x2005FFFF into sram=0x10002800-0x1000FFFF Loaded process[0] from flash=0x20030000-0x200304E3 into sram=0x10005000-0x10006703 = "c_hello" OpenTitan initialisation complete. Entering main loop [0] function_call @0x200300a8(0x20030080, 0x10005000, 0x1704, 0x10005000) [0] memop(0, 0x100059dc) = Success [0] memop(10, 0x10005800) = Success [0] memop(11, 0x100059dc) = Success [0] read-only allow(0x1, 1, @0x10005800, 0xe) = AllowReadOnlySuccess(0x0, 0) [0] remove_pending_upcalls[0x1:1] = 0 upcall(s) removed [0] subscribe(0x1, 1, @0x200300fc, 0x0) = SubscribeSuccess(0x0, 0) [0] cmd(0x1, 1, 0xe, 0x0) = Success [0] yield. which: 1 Hello World! [0] schedule[0x1:1] @0x200300fc(0xe, 0x0, 0x0, 0x0) = Ok(()) [0] function_call @0x200300fc(0xe, 0x0, 0x0, 0x0) [0] yield. which: 1

So next I'm trying to load it at 0x200304E4 what gives:

App flash does not match requested address. Actual:0x20030564, Expected:0x20030880

I00001 boot_rom.c:76] Boot ROM initialisation has completed, jump into flash! Loading processes from flash=0x20030000-0x2005FFFF into sram=0x10002800-0x1000FFFF Loaded process[0] from flash=0x20030000-0x200304E3 into sram=0x10005000-0x10006703 = "c_hello" Error loading processes! App flash does not match requested address. Actual:0x20030564, Expected:0x20030880 OpenTitan initialisation complete. Entering main loop [0] function_call @0x200300a8(0x20030080, 0x10005000, 0x1704, 0x10005000) [0] memop(0, 0x100059dc) = Success [0] memop(10, 0x10005800) = Success [0] memop(11, 0x100059dc) = Success [0] read-only allow(0x1, 1, @0x10005800, 0xe) = AllowReadOnlySuccess(0x0, 0) [0] remove_pending_upcalls[0x1:1] = 0 upcall(s) removed [0] subscribe(0x1, 1, @0x200300fc, 0x0) = SubscribeSuccess(0x0, 0) [0] cmd(0x1, 1, 0xe, 0x0) = Success Hello World! [0] schedule[0x1:1] @0x200300fc(0xe, 0x0, 0x0, 0x0) = Ok(()) [0] yield. which: 1 [0] function_call @0x200300fc(0xe, 0x0, 0x0, 0x0) [0] yield. which: 1

Can I somehow deduce address where to load second app in QEMU?

[alistair23]

There is also a fork of QEMU for ESP32 support. So we could test the Tock ESP board in QEMU as part of the CI: https://github.com/esp32-open-mac/qemu/blob/esp-develop/hw/riscv/esp32c3.c