`v0.11`: Redesign config and build system, split into sub-crates, and provide disk image builder

[phil-opp]

This pull requests implements the next version of the bootloader crate: v0.11. It contains multiple breaking changes:

• Separate API crate: The bootloader is now split into two parts: An API crate to make kernels loadable by the bootloader and the actual bootloader implementation. This makes the build process for kernels much easier and faster.
• New config system: Instead of configuring the bootloader via a special table in the Cargo.toml, the configuration now happens through a normal Rust struct, which is part of the entry_point! macro. The macro then serializes the config struct at compile time and places it in a special ELF output section. The compile time serialization happens through a manually implemented const fn of the config struct.
• Load the kernel at runtime: The bootloader is now able to load files from FAT partitions at runtime. Thus, we don't need to link the kernel into the bootloader executable anymore. As a result, we don't need to recompile the bootloader on kernel changes anymore. We also load the config at runtime from the kernel's ELF section, which eliminates the second reason for recompiling the bootloader as well.
• Split into sub-crates: Since the bootloader build process does not need access to the kernel executable or its Cargo.toml anymore, we can build the different parts of the bootloader independently. For example, the BIOS boot sector is now a separate crate, and the UEFI bootloader is too.
• Library to create disk images: To create an abstraction the complex build steps of the different bootloader executables, we compile them inside cargo build scripts. At the top level, we provide a bootloader library crate, which compiles everything as part of its build script. This library includes functions for creating BIOS and UEFI disk images for a given kernel. These functions can be used e.g. from a builder crate or a build script of the downstream operating system.

Taken together, these changes should make the bootloader crate much easier to use and also more accessible to contributors.

---

This PR is still a work in progress. There are still many things missing, including:

• [x] The BIOS implementation
  • [x] Integrate into build system
  • [x] Pure-Rust boot sector (16-bit real mode) that loads the second stage from a ~FAT partition~ MBR partition entry
  • [x] Pure-Rust second stage (16-bit unreal mode) that initializes the CPU and loads the kernel and third stage from the FAT partition
    • [x] Parse the FAT partition
    • [x] Locate the kernel-x86_64 file in the root folder
    • [x] Set up protected or unreal mode
    • [x] Load kernel to protected mode memory
    • [x] Create memory map and pass it to stage 3
    • [x] Set up framebuffer
      • [x] search VESA modes for a suitable one
      • [x] fill in framebuffer info in boot information
      • [x] enable the chosen VESA mode
      • [ ] consider config of kernel ELF (maybe later, not clear how we can read the config yet)
    • [x] Load third stage and jump to it
  • [x] Stage 3 (protected mode)
    • [x] Use framebuffer logger instead of VGA text mode
    • [x] Set up long mode and identity mapping
      • [x] Identity mapping
      • [x] Compatibility mode
      • [x] 64-bit mode
    • [x] Jump to stage 4
  • [x] Stage 4 (long mode)
    • [x] link with the common crate to handle the config and map the kernel
• [x] Adjust README for new version
• [ ] Create migration guides
  • [ ] From version 0.9.x
  • [ ] From version 0.10.x
• [x] Clean up the repo and remove all legacy files
• [ ] For the kernel-requested physical memory mapping we don't want to skip regions. However, instead of mapping the full 0..max_phys_addr range, we could instead iterate over the reported regions and map only them. See #259 for details.
• [x] Fix BIOS boot error on real machine caused by VESA: https://github.com/rust-osdev/bootloader/pull/232#issuecomment-1243004273
• [x] Improve error message when kernel has no bootloader-config section
• [x] Try to find a way to keep the bootloader-config section when lto is enabled (see https://github.com/rust-osdev/bootloader/pull/232#issuecomment-1244696577)

[Freax13]

I've been working on a PXE bootloader based on the this branch. @phil-opp Would you prefer for me to create the pull request before or after you've finished working on this branch?

[phil-opp]

@Freax13 Sounds great!

Would you prefer for me to create the pull request before or after you've finished working on this branch?

Both are fine, so it depends on the state of your code. If it's ready already, feel free to send a PR! I just don't want to block the v0.11 release any longer than necessary.

[kennystrawnmusic]

So the BIOS implementation seems to need a lot of work done on it, but what about the UEFI implementation in this redesign? Less to do there?

[phil-opp]

@kennystrawnmusic The UEFI implementation is already done. It's much easier to load the kernel and the memory map there.

[Freax13]

Now that the bootloader is independent from the kernel, we could compile the bootloader in CI and publish it that way. This would also make it easier for people who don't want to use bootloader::create_boot_partition, bootloader::create_uefi_disk_image etc and want to use their own build scripts instead. This might be out of the scope for this pr though.

[phil-opp]

Thanks a lot for taking the time to review! I know that this PR is huge and probably a nightmare to review. Fortunately, the control flow is quite linear, so we mostly need to check whether it is able to load different kernels and whether the given boot information is correct. Maybe it's a good idea to pre-publish an alpha release for this PR and try it out on different projects that are currently using bootloader v0.9 and v0.10?

Now that the bootloader is independent from the kernel, we could compile the bootloader in CI and publish it that way. This would also make it easier for people who don't want to use bootloader::create_boot_partition, bootloader::create_uefi_disk_image etc and want to use their own build scripts instead. This might be out of the scope for this pr though.

Interesting idea! I agree that there is definitely some value in creating GitHub releases with precompiled artifacts. However, the current form of the BIOS bootloader might make that difficult since it consists of many indepentent executables that have strict requirements about the partition and file system layout. So it's probably better to do this in a follow-up PR and maybe improve the BIOS setup beforehand.

[phil-opp]

The CI errors should be fixed by https://github.com/rust-lang/rust/pull/101413, which should be part of the next Rust nightly.

[Freax13]

Thanks a lot for taking the time to review! I know that this PR is huge and probably a nightmare to review. Fortunately, the control flow is quite linear, so we mostly need to check whether it is able to load different kernels and whether the given boot information is correct.

I'll try to do another round of review today.

Maybe it's a good idea to pre-publish an alpha release for this PR and try it out on different projects that are currently using bootloader v0.9 and v0.10?

Github codesearch can be used to find projects using the bootloader: "bootloader = " ("0.9" OR "0.10") path:Cargo.toml I've been using a fork of this branch for my own project for a while now (the fork was needed because the bootloader still relied on bindeps and that doesn't work with packages published to crates.io or used with git). I'll switch back to this branch and test.

[Freax13]

I'll also test on some real platforms.

[phil-opp]

Github codesearch can be used to find projects using the bootloader: "bootloader = " ("0.9" OR "0.10") path:Cargo.toml

Good idea!

I've been using a fork of this branch for my own project for a while now (the fork was needed because the bootloader still relied on bindeps and that doesn't work with packages published to crates.io or used with git). I'll switch back to this branch and test.

Ah, I wasn't aware of that. I think the only remaining uses of bindeps are for the test framework, so it should no longer be a problem. But if you encounter any issues, please let me know!

I'll also test on some real platforms.

That would be great, I haven't done that at all yet.

[Freax13]

Hmm, I'm getting a crash on one of my NUCs when booting with the UEFI bootloader:

!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000  I:0 R:0 U:0 W:0 P:0 PK:0 SS:0 SGX:0
RIP  - 00000000340E964F, CS  - 0000000000000038, RFLAGS - 0000000000010287
RAX  - 0000000034107219, RCX - FFFFFFFD00000000, RDX - 00000000417FFB08
RBX  - 0000000000000000, RSP - 00000000340E7360, RBP - 0000000000000FFF
RSI  - 0000000000000000, RDI - 00000000001536D3
R8   - 0000000034107B88, R9  - 00000000417EC000, R10 - 0000000041588018
R11  - 0000000034108B90, R12 - 00000000340E7650, R13 - 00000000340E73F0
R14  - 8000000000000002, R15 - 0000000000001340
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010013, CR2 - FFFFFFE834107219, CR3 - 0000000033A01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000416CCA98 0000000000000047, LDTR - 0000000000000000
IDTR - 0000000036C43018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 00000000340E6FC0
!!!! Can't find image information. !!!!

I'll investigate what's going on.

[Freax13]

The bounds check for the logger look wrong: https://github.com/rust-osdev/bootloader/blob/960671c3669c7d1be95f1dfb40afdf9bea8172f8/common/src/logger.rs#L103-L110 It's mixing up width and height. I've ran into index out of bounds panics because of this.

[Freax13]

Hmm, I'm getting a crash on one of my NUCs when booting with the UEFI bootloader:

!!!! X64 Exception Type - 0E(#PF - Page-Fault)  CPU Apic ID - 00000000 !!!!
ExceptionData - 0000000000000000  I:0 R:0 U:0 W:0 P:0 PK:0 SS:0 SGX:0
RIP  - 00000000340E964F, CS  - 0000000000000038, RFLAGS - 0000000000010287
RAX  - 0000000034107219, RCX - FFFFFFFD00000000, RDX - 00000000417FFB08
RBX  - 0000000000000000, RSP - 00000000340E7360, RBP - 0000000000000FFF
RSI  - 0000000000000000, RDI - 00000000001536D3
R8   - 0000000034107B88, R9  - 00000000417EC000, R10 - 0000000041588018
R11  - 0000000034108B90, R12 - 00000000340E7650, R13 - 00000000340E73F0
R14  - 8000000000000002, R15 - 0000000000001340
DS   - 0000000000000030, ES  - 0000000000000030, FS  - 0000000000000030
GS   - 0000000000000030, SS  - 0000000000000030
CR0  - 0000000080010013, CR2 - FFFFFFE834107219, CR3 - 0000000033A01000
CR4  - 0000000000000668, CR8 - 0000000000000000
DR0  - 0000000000000000, DR1 - 0000000000000000, DR2 - 0000000000000000
DR3  - 0000000000000000, DR6 - 00000000FFFF0FF0, DR7 - 0000000000000400
GDTR - 00000000416CCA98 0000000000000047, LDTR - 0000000000000000
IDTR - 0000000036C43018 0000000000000FFF,   TR - 0000000000000000
FXSAVE_STATE - 00000000340E6FC0
!!!! Can't find image information. !!!!

I'll investigate what's going on.

I'm pretty sure this crash is caused by the custom UEFI memory type we use for the kernel. The address in CR2 seems to be linearly dependent on the provided memory type. There used to be a bug in edk2 that stopped those custom memory types working, but that bug was fixed more than 10 years ago in https://github.com/tianocore/edk2/commit/10fe0d814add860a1040e648b1f5782c0de350e6. I don't know why this bug would be present on platforms I've acquired less than a year ago. Just to be sure I checked grub and they also seem to be using the standard memory type LOADER_DATA (https://github.com/jiazhang0/grub-efi/blob/cb3fa40dae2cd397e63bd0b1195fec085564ed7e/efi/efimm.c#L91-L95). I also found someone else coming across the same issue. Changing the memory type to LOADER_DATA fixed the crash.

[Freax13]

Changing the memory type to LOADER_DATA fixed the crash.

With that patched I was able to test the UEFI bootloader on 6 platforms (3 Intel, 3 AMD) and it worked on all of them.

[Freax13]

Changing the memory type to LOADER_DATA fixed the crash.

With that patched I was able to test the UEFI bootloader on 6 platforms (3 Intel, 3 AMD) and it worked on all of them.

I've published the kernel I used for testing under https://github.com/Freax13/test-kernel.

[phil-opp]

Thanks a lot for testing and debugging this!

It's mixing up width and height. I've ran into index out of bounds panics because of this.

Fixed in https://github.com/rust-osdev/bootloader/pull/232/commits/47a4c85d96e285f8b208142e455a04eb0a61bcf6.

Changing the memory type to LOADER_DATA fixed the crash.

Interesting! So we need to find another way to recognize the bootloader memory region after allocation, right?

[Freax13]

Interesting! So we need to find another way to recognize the bootloader memory region after allocation, right?

Yes, but do we even need to mark LOADER_DATA regions are usable? AFAICT we only use LOADER_DATA for the kernel and the memory map and we don't want to mark either of those as usable after exiting boot services. Unless I'm missing something, us marking the memory map as usable is a bug, isn't it?

[phil-opp]

Yes, but do we even need to mark LOADER_DATA regions are usable? AFAICT we only use LOADER_DATA for the kernel and the memory map and we don't want to mark either of those as usable after exiting boot services. Unless I'm missing something, us marking the memory map as usable is a bug, isn't it?

I think loader data is also used for the UEFI application itself, i.e. storing the data sections of the bootloader executable. The memory map is converted later to a common format and copied into a new region. So we don't need the UEFI-specific memory map anymore after switching to the kernel.

[Freax13]

Ah that explains it.

I guess we need something similar to https://github.com/rust-osdev/bootloader/blob/130d5f0ac8b3cd340ea25ce3306068db48112f70/common/src/legacy_memory_region.rs#L113-L130 for the kernel. Damn the custom memory type would have been so much cleaner.

[phil-opp]

I guess we need something similar to [...]

Done in https://github.com/rust-osdev/bootloader/pull/232/commits/08e4b5829bf5882d9d396e641e32b65de72704b2

[Freax13]

https://github.com/rust-osdev/bootloader/blob/667e57f552e214f9c19848306e03b00d91a8114f/api/src/info.rs#L175-L177 This is unsound because create_buffer creates a mutable reference which is not allowed to alias other references created by buffer().

[phil-opp]

Thanks a lot a for reviewing!

I try to create add a basic migration guide later, then I think we can merge :tada:.

[asensio-project]

Hello,

I tried this version with the test-kernel from @Freax13, with UEFI works fine. But with legacy BIOS on bare metal doesn't work. I get a PANIC on bios/stage-2/src/main.rs on the line 111:10, the cause is an unwrap on a None value of an Option.

Thanks for this amazing project.

[phil-opp]

@asensio-project Thanks for testing! Seems like this is related to the VESA config here:

https://github.com/rust-osdev/bootloader/blob/f95e9a97190c118f94f914b0c2483b137b8be39d/bios/stage-2/src/main.rs#L108-L111

I'll look into it!

[TheBotlyNoob]

getting

UEFI bootloader started; trying to load kernel
panicked at 'called `Option::unwrap()` on a `None` value', common\src\lib.rs:67:27

It seems to be coming from https://github.com/rust-osdev/bootloader/blob/next/common/src/lib.rs#L67-L69 and, only happens when I compile in release mode with lto = true

[toothbrush7777777]

It seems to be coming from https://github.com/rust-osdev/bootloader/blob/next/common/src/lib.rs#L67-L69 and, only happens when I compile in release mode with lto = true

@TheBotlyNoob You probably need a custom linker script specifying to KEEP those custom sections.

[phil-opp]

@TheBotlyNoob Thanks for reporting! There was a bug in the entry_point macro so that the config was not marked as used properly. I pushed a fix in 076bea9 and added a test to prevent regressions.

[phil-opp]

Seems like this is related to the VESA config here:

https://github.com/rust-osdev/bootloader/blob/f95e9a97190c118f94f914b0c2483b137b8be39d/bios/stage-2/src/main.rs#L108-L111

I'll look into it!

Whew, that took a lot of debugging. It turns out that there were two bugs in the VESA code related to segmented memory. Apparently QEMU doesn't use segments in that case, so it was only triggered by real hardware. I pushed fixes for the bugs in https://github.com/rust-osdev/bootloader/pull/232/commits/6eb0abbb7e5b4b6088fb49452c7dc976183a946a and https://github.com/rust-osdev/bootloader/pull/232/commits/a6d69e9b4bd9daf4dc006d25b9d11d6c06edaab2. Now it boots fine with both UEFI and BIOS on my old laptop.

[Freax13]

Just did another test run: Tested the UEFI bootloader on 6 platforms (4x Intel, 2x AMD) and the BIOS bootloader on 1 platform (Intel). all successful

[asensio-project]

I tried on x86_64 bare metal, works on UEFI and BIOS. Thanks!