seL4
boot: allow early printf() - before kernel vspace
On ARM and RISC-V (not x86, which uses IO ports), the kernel's printf() call goes via the virtual address, which means that early prints before we setup virtual memory cause faults. Any issues at early stages will silently fail, e.g. various assert() that may go off.
We add a global variable that gets adjusted once the vspace exists so that the UART drivers can go through the physical memory address (since the kernel expects to be mapped 1:1 at boot time) early-on and then migrate to virtual address later. This makes debugging early-boot issues significantly easier on bare metal (without gdb).
Fixes: #540 Fixes: #1191
No idea why ZYNQ7000_debug_clang_32 fails, but that needs to resolved first before merging this:
Enabling MMU and paging
Warning: Could not infer GIC interrupt target ID, assuming 0.
ERROR: DTB at [40000..42e47) exceeds PADDR_TOP (1ff00000)
seL4 called fail at /github/workspace/kernel/src/arch/arm/kernel/boot.c:674 in function init_kernel, saying "ERROR: kernel init failed"
Bingo...
ELF-loading image 'kernel' to 0
paddr=[0..3ffff]
vaddr=[e0000000..e003ffff]
virt_entry=e0000000
Kernel is loaded to 0x_e000_0000, but this also happens to be the UART address: https://github.com/seL4/seL4/blob/09e6c3f5e27607cd6f7854bdc5e87d829c48d500/tools/dts/zynq7000.dts#L260-L269
But for some reason the kernel identity-maps everything below the kernel first vaddr as identity, but everything above is direct mapped? Somehow I've become mistaken, I thought the kernel booted entirely identity-mapped (just... in virtual memory) until activate_kernel_vspace() happened. I guess we were just lucky that none of the platforms with UART above the kernel didn't happen to write into an address that was direct-mapped into non-memory, somehow. I guess this PR isn't entirely valid... only when the UART is below the kernel first vaddr. I don't understand why...
https://github.com/seL4/seL4_tools/blob/26f94df7e97a04f8ee49c1e94a0aa12036581f71/elfloader-tool/src/arch-arm/32/mmu.c#L13-L50
Does that mean zync7000's seL4_UserTop 0xe0000000 is ~wrong~ unfortunate? On 32-bit it's a bit crowded in the address space because of the limited range.
Kernel is loaded to
0x_e000_0000, but this also happens to be the UART address:https://github.com/seL4/seL4/blob/09e6c3f5e27607cd6f7854bdc5e87d829c48d500/tools/dts/zynq7000.dts#L260-L269
Good find.
Somehow I've become mistaken, I thought the kernel booted entirely identity-mapped (just... in virtual memory) until
activate_kernel_vspace()happened.
No, that's how Efloader boots.
The current kernel can't boot with identity mapping, that's why it needs Elfloader. There is no way other than making all code position independent to make that work for all kernel code in its current form (because the same code would be executed with different addresses, same for data accesses). So Elfloader needs to map the kernel in the same way as the kernel will map its memory in activate_kernel_vspace(). Basically doing more or less the same thing twice. This is one of many reasons why I want to make seL4 self-booting without Elfloader. Currently this is a big part of the kernel boot protocol which is entirely undocumented.
I guess we were just lucky that none of the platforms with UART above the kernel didn't happen to write into an address that was direct-mapped into non-memory, somehow.
Elfloader has an if-check to handle the case where the UART is in memory overlapping with the kernel AFAIK, to prevent this from happening. The kernel didn't do printf until activate_kernel_vspace(), and at that point it got mapped to a sane virtual address.
I guess this PR isn't entirely valid... only when the UART is below the kernel first vaddr. I don't understand why...
I think this PR makes the wrong assumption that there always is a 1:1 mapping for the physical UART address pre activate_kernel_vspace(). Might be enough to undo the whole idea, except if you find some work-around.
What about having the uart mapped to a defined virtual address as part of the boot contract with the elfloader. The elfloader always has uart working when it switches to the kernel, and the elfloader sets up mappings for the kernel, so the elfloader could always map the uart registers to a compatible virtual address the kernel can use.
Yes, I think that might be a good idea. It might require the elfloader to know about where the kernel expects UART to be mapped, but that's probably easy enough to do...
What about having the uart mapped to a defined virtual address as part of the boot contract with the elfloader. The elfloader always has uart working when it switches to the kernel, and the elfloader sets up mappings for the kernel, so the elfloader could always map the uart registers to a compatible virtual address the kernel can use.
If doing this, the loader should map the UART to the final location the kernel will use after boot. And then there are no changes on the kernel side needed.
I think this is fine to do to make debugging easier, but I do not think the kernel should rely on the loader doing this correctly. Or the loader has a way to tell the kernel it mapped the UART or something. The boot protocol between the loader and the kernel is already obscure and badly documented enough, this would add another subtle requirement.
Edit: Perhaps add a kernel config option (default off, enabled when building with a compatible loader) and introduce early_printf and use that pre-init.