heads
heads copied to clipboard
linuxboot
Add Talos 2 board (OpenPower)
This provides a configuration for OpenPower that builds (wasn't fully tested yet). It seems to be the first non-x86 target for Heads and might be the time to think how you want to support multiple targets.
This isn't a finished PR and the point is to request comments. Below are some suggestions which you might want to consider.
I haven't tried to enable all modules. I think this is a typical config with some things disabled. I'm not sure which are necessary. Might enable and fix build of other things. At the moment at least 31 out of 48 available modules build fine.
Building for power produces zImage instead of bzImage. The latter doesn't work (no rule to make bzImage). Image name is configurable now.
For coreboot, I specifed -b in coreboot_repo to switch to the branch. Might want to improve this.
For now I hoisted everything to board configuration via CONFIG_<MODULE>_TARGET variables and I don't really like how it looks (naming is very inconsistent). Maybe you want to have a generic CONFIG_TARGET, which will then be translated within each module and fail for unknown targets? This might ease adding new targets in the future thanks to more obvious errors.
@SergiiDmytruk did a regression build against https://github.com/osresearch/heads/pull/1002/commits/7efe7cecc6c213a27c8d6f037293a1f15de1acae and it was successful.
Everything x86 builds as expected from last commit. I would recommend following your branch from CircleCI so that each commit pushed triggers a CircleCI build for that commit.
@SergiiDmytruk
You could change this for something similar to the following and tune to have your boards built first with save_cache. (build_and_persist is costy since saving workstpace and restoring workspaces). You might want to only build talos_2 and talos_2_lite and wipe the rest to speed up testing also if regression testing is not needed in all builds (I think changes that are coming next should probably only happen in linux and coreboot configs and trees, so touching other x86 and testing for regression might not be needed.
# Below, sequentially build one board for each coreboot
# version. The last board in the sequence is the dependency
# for the parallel boards built at the end, and also save_cache.
#Coreboot Talos
- build_and_persist:
name: talos_2
target: talos_2
requires:
- prep_env
# Coreboot 4.8.1
- build_and_persist:
name: qemu-coreboot
target: qemu-coreboot
requires:
- prep_env
# Coreboot 4.11
- build_and_persist:
name: kgpe-d16_workstation
target: kgpe-d16_workstation
requires:
- qemu-coreboot
# Coreboot 4.13
- build_and_persist:
name: librem_mini
target: librem_mini
requires:
- kgpe-d16_workstation
- save_cache:
requires:
- librem_mini
#
#
# Those onboarding new boards should add their entries below.
#
#
- build:
name: talos_2_le
target: talos_2_le
requires:
- talos_2
- build:
name: x220-hotp-maximized
target: x220-hotp-maximized
requires:
- librem_mini
- build:
name: x220-maximized
target: x220-maximized
requires:
- librem_mini
- build:
name: t420-hotp-maximized
target: t420-hotp-maximized
requires:
- librem_mini
- build:
name: t420-maximized
target: t420-maximized
requires:
- librem_mini
- build:
name: x230-flash
target: x230-flash
requires:
- librem_mini
- build:
name: t430-flash
target: t430-flash
requires:
- librem_mini
- build:
name: t430
target: t430
requires:
- librem_mini
- build:
name: x230
target: x230
requires:
- librem_mini
- build:
name: x230-hotp-verification
target: x230-hotp-verification
requires:
- librem_mini
- build:
name: x230-maximized
target: x230-maximized
requires:
- librem_mini
- build:
name: t430-hotp-maximized
target: t430-hotp-maximized
requires:
- librem_mini
- build:
name: x230-hotp-maximized
target: x230-hotp-maximized
requires:
- librem_mini
- build:
name: t430-maximized
target: t430-maximized
requires:
- librem_mini
- build:
name: qemu-coreboot-fbwhiptail
target: qemu-coreboot-fbwhiptail
requires:
- librem_mini
- build:
name: kgpe-d16_workstation-usb_keyboard
target: kgpe-d16_workstation-usb_keyboard
requires:
- librem_mini
- build:
name: kgpe-d16_server
target: kgpe-d16_server
requires:
- librem_mini
- build:
name: kgpe-d16_server-whiptail
target: kgpe-d16_server-whiptail
requires:
- librem_mini
- build:
name: librem_l1um
target: librem_l1um
requires:
- librem_mini
- build:
name: librem_mini_v2
target: librem_mini_v2
requires:
- librem_mini
Or without x86 regression testing this simpler version:
# Below, sequentially build one board for each coreboot
# version. The last board in the sequence is the dependency
# for the parallel boards built at the end, and also save_cache.
#Coreboot Talos
- build_and_persist:
name: talos_2
target: talos_2
requires:
- prep_env
- save_cache:
requires:
- talos_2
#
#
# Those onboarding new boards should add their entries below.
#
#
- build:
name: talos_2_le
target: talos_2_le
requires:
- talos_2
@tlaurion, thanks. I left one x86 target to see if CI works in this case. I'm not sure because it's not enough to just replace build/ to switch targets, need to also remove install/ and crossgcc/ or they will mess up the build (libraries and includes will be used from wrong toolchain).
@tlaurion, thanks. I left one x86 target to see if CI works in this case. I'm not sure because it's not enough to just replace
build/to switch targets, need to also removeinstall/andcrossgcc/or they will mess up the build (libraries and includes will be used from wrong toolchain).
@SergiiDmytruk
I see.... x86 centric and completely understandable until now :/
Probably better to fix INSTALL by appending TARGET/arch suffix to that directory (while musl-cross path should deal with x86 and power from inclusion of the module def) from now on.
Fixating caches to include those new directories into them would also be required to be reused:
- Include Makefile so that caches digests (tag) are not reused when Makefile changes (that didn't change for a while and wasn't included)
- Names for cache save/restore can be reused without change (above will change digest, so caches will not be saved/restored if digest changed)
- Where cache directories saved/restored should be changed to match new directories created having TARGET (or arch) appended
@SergiiDmytruk your board dir and board names are matching to "talos-2" and "talos-2-le" not "talos_2" and "talos_2_le" which you fixed under https://github.com/osresearch/heads/blob/61002eb088c5819867e12dcdb0d5591bc33a9aca/.circleci/config.yml
The last build was manually cancelled, but should have continued the build!
@tlaurion It wasn't me:
We have detected an anomaly that violates the CircleCI Terms of Service.
We did not test this push because the project has been suspended. Please contact us if you think this is a mistake.
@SergiiDmytruk weird. This is an error. Building here meanwhile, but yes, this is open source project so should not be a violation of their terms of services.
Let's see if that build is also suspended.
@SergiiDmytruk note that old cache is being reused for last successful build with same modules and patches created digests as explained in previous post. Those are saved into workspace now and will be passed to board builds to be reused, so it will probably fail since ./install and ./crossgcc (and build/musl-cross) will be shared between x86 and power.
Builds taking more then 3 hours per task are also cancelled per terms and services, which is why we struggle using CIs for Heads.
@tlaurion CI works again for me after contacting support. Haven't updated how caching is done yet.
@tlaurion Currently Heads produces kernel and initrd separately. Was embedding of initrd into the kernel ever considered? We probably won't need this for coreboot (skiroot needs it), but I thought I'll bring it up just in case there were attempts and known issues. At the moment I only see this way:
- Create empty
initrd.cpio - Build everything including real initrd
- (Re)link kernel
@tlaurion Currently Heads produces kernel and initrd separately. Was embedding of initrd into the kernel ever considered? @SergiiDmytruk never thought about it....
initrd is actually what Heads is (outside of the patches applied to linux kernel and patches applied to coreboot (less and less, but still.) Just to make sure, you can see those compressed artifacts under /build/$BOARD dir and under CI. Example artifacts for a built board
- bzImage (kernel)
- hashes.txt (contains hashes of all compiled and bundled steps to be able to compare between builds)
- final rom for board (coreboot+ kernel + initrd per coreboot last step of putting everything together)
- heads.cpio (scripts under /initrd)
- modules.cpio (linux kernel modules per linux heads's module)
- tools.cpio (modules bin, modules libs)
- initrd.cpio.xz emcompassing dev.cpio, then modules.cpio, tools.cpio and heads.cpio above.
Some reverse understanding is currently possible since initrd is built separately: tools.cpio, modules.cpio initrd.cpio and heads.cpio are bundled under initrd.cpio.xz. The more we bundle things the more difficult it is for newcomers to reverse the bundling inspect and diagnose reproducibility problems, but that might be only a documentation guide away. I do not have a preference over this, if you think the same outcome can be achieved, while understanding what is happening here.
We probably won't need this for coreboot (skiroot needs it), but I thought I'll bring it up just in case there were attempts and known issues. At the moment I only see this way:
1. Create empty `initrd.cpio` 2. Build everything including real initrd 3. (Re)link kernel
Let me know if this idea is still applicable following details provided above.
@tlaurion, thanks for the detailed explanation.
Let me know if this idea is still applicable following details provided above.
I think everything should be OK as is once coreboot will be able to use Heads as payload.
I think everything should be OK as is once coreboot will be able to use Heads as payload.
I do not think it will be in the near future. The goal for now is to replace skiroot with heads (while still using skiboot as the only available coreboot payload). As defined here: https://github.com/osresearch/heads/issues/720#issuecomment-867522437
We won't be able to start heads kernel as a coreboot Linux payload as it is currently done for other platforms. At least for now, we only can start skiboot from coreboot. We need it for OPAL. Then, the skiboot can load Linux - in this case it could load heads kernel and initramfs.
@tlaurion I've re-committed changes and marking this PR for review (but will see if CI passes with all the boards). Below is a summary of the changes since the previous comment.
The build eventually started to fail due to mixing of files of different architectures under build/, I looked into updating makefiles of modules, but it seemed unfeasible (very specific to each project and would add many more patches), so now it's build/<arch>/<module|board> with gawk being build in build/ once. Hopefully, I updated configs and CircleCI configuration right.
It's just one board again with BE coreboot and LE everywhere else, which seems to be typical for PowerPC.
Because we need skiboot, Heads can now optionally build bundled Linux kernel and the new board asks for it.
Patches for coreboot can be dropped after https://github.com/3mdeb/coreboot/pull/79 is merged.
There is now CONFIG_TARGET (x86 by default) and modules which had build issues branch on it to determine host triplet instead of hard-coding this module-specific knowledge in board definition.
I think everything should be OK as is once coreboot will be able to use Heads as payload.
@SergiiDmytruk Not sure about the next steps here. Do we wait to merge once Heads can be used as a payload?
I see from here:
Uploading /root/project/build/ppc64/talos-2 to build/ppc64/talos-2
Uploading /root/project/build/ppc64/talos-2/hashes.txt (9.8 kB): DONE
Uploading /root/project/build/ppc64/talos-2/heads-talos-2-v0.2.0-1047-g586ddd0.rom (2.1 MB): DONE
Uploading /root/project/build/ppc64/talos-2/heads.cpio (167 kB): DONE
Uploading /root/project/build/ppc64/talos-2/initrd.cpio (15 MB): DONE
Uploading /root/project/build/ppc64/talos-2/initrd.cpio.xz (3.8 MB): DONE
Uploading /root/project/build/ppc64/talos-2/logs.tar.gz (1.2 MB): DONE
Uploading /root/project/build/ppc64/talos-2/modules.cpio (560 kB): DONE
Uploading /root/project/build/ppc64/talos-2/tools.cpio (14 MB): DONE
Uploading /root/project/build/ppc64/talos-2/zImage (7.9 MB): DONE
Uploading /root/project/build/ppc64/talos-2/zImage.bundled (12 MB): DONE
Where coreboot image doesn't seem to bundle much. Can you detail the current state of the build?
@tlaurion
Do we wait to merge once Heads can be used as a payload?
It's not clear when that will be the case (i.e., when coreboot will support OPAL). A much closer target is the ability to start skiboot, although I don't know its exact status (might work already, not much changes on the branch) and when it will make its way upstream.
Where coreboot image doesn't seem to bundle much. Can you detail the current state of the build?
It's small because its payload is skiboot. zImage.bundled should be inserted into flash separately (with pflash commands like here, the alternative is flashing everythin in one go with a full image composed locally) and skiboot will pick it up from there. coreboot needs to be installed the same way. Actually we need .signed.ecc ROM as well as bootblock.signed.ecc in this case. I'll change that and also test all build artifacts together, thanks.
Not sure about the next steps here.
My concern about this PR is mostly about changes to the build system, which is the bulk of it. Do they look OK or something needs to be done differently?
As a basic regression test, I flashed https://113-380596387-gh.circle-artifacts.com/0/build/x86/x230-hotp-maximized/heads-x230-hotp-maximized-v0.2.0-1047-g586ddd0.rom internally and booted Tails with detached signature file verified successfully.
My concern about this PR is mostly about changes to the build system, which is the bulk of it. Do they look OK or something needs to be done differently?
@SergiiDmytruk: I will review the changes done on the build system, but I do not see anything problematic and it opens the door for integration of other architectures then x86.
One consideration for Heads (usability) is to use gui-init instead of generic-init, which dependencies are outlined when comparing qemu-coreboot and qemu-coreboot-fbwhiptail (which requires functional graphic framebuffer under linux config). That means that CAIRO and FBWHIPTAIL modules and dependencies might cause you some problems along the way for a workstation aimed build, but I think this would be the desired outcome of the board to be functionally the same as other boards also testing gui-init implemented policy vs the old generic-init which is less used (and mostly deprecated).
Note that for the qemu-coreboot board config here, one could simply replace generic-init script with gui-init and have whiptail do the prompts on the console without fb (uglier) but would also work, in case no graphical framebuffer is chosen/possible (that would also work remotely over console, just like for the kgpe-d16 boards configs what were made: server (whiptail) vs workstation (fbwhiptail), where server is aimed to be accessed over the BMC console for remote attestation).
@MrChromebox: See any change you do not like in the build system?
@tlaurion Switched the board to use gui-init with whiptail in console mode. Tested GUI mode and it works fine, but it's much less convenient to use remotely.
Also flashed coreboot and kernel built by Heads, but skiboot hanged on CPU initialization. Something is probably not finished yet.
@tlaurion Switched the board to use
gui-initwithwhiptailin console mode. Tested GUI mode and it works fine, but it's much less convenient to use remotely.
@SergiiDmytruk Can you elaborate on "less convenient"? Not scriptable for sure unless doing screen captures and automatic input for automatic tests. :/
But gui-init is what is currently used by Purism, Nitrokey and Insurgo. generic-init is kind of Heads legacy base, which was used prior of HOTP support (joint effort between Heads, Nitrokey and Purism) and is not much used anymore compared to gui-init codebase. But might be good enough in the current state of things.
@tlaurion
Can you elaborate on "less convenient"?
I use it either via SSH or through a Web-interface. And OpenBMC doesn't seem to accept input where it shows the screen in the browser, so it has to be provided in a separate window where it shows textual console. That's all I meant, text is more universal. That said, how it will look finally doesn't matter too much, because it's configurable, so I can enable CAIRO and FBWHIPTAIL if you like it more.
But gui-init is what is currently used by Purism, Nitrokey and Insurgo.
I think you're trying to justify switching to gui-init, which I already did (works fine). Just not in a graphical mode.
I think you're trying to justify switching to gui-init, which I already did (works fine). Just not in a graphical mode.
Graphical mode (FBWHIPTAIL, cairo and other dependencies) can be switched on for workstation builds later on. My advice here was to create other board configs for workstation (FBWHIPTAIL)/server (WHIPTAIL) with different subsets of dependencies tested so that building those is also confirmed functional.
I understand what you are saying. qemu-coreboot-fbwhiptail is slow. kgpe-d16 server is using whiptail for BMC, not fbwhiptail. Sorry for the noise.
@tlaurion the only thing I see that's less than ideal is adding another subdir (arch) that needs to be accounted for in the coreboot configs, adding another ../ in the payload and initrd paths.
I think the git repo handling could be done better, perhaps specifying a commit hash vs a branch, since the former is immutable and the latter is not. I have a patch in Purism's tree to do that, but can't be used with --depth=1 in the git clone so there's room for improvement in my current implementation
@MrChromebox
the only thing I see that's less than ideal is adding another subdir (arch) that needs to be accounted for in the coreboot configs, adding another ../ in the payload and initrd paths.
I didn't like that editing either. Have you considered doing automake-like expansion in configs? Something like
CONFIG_PAYLOAD_FILE="../../../build/x86/librem_l1um/bzImage"
would become
CONFIG_PAYLOAD_FILE="@BOARD_DIR@/bzImage"
sed should be enough to do such simple replacements during copying of a config.
@SergiiDmytruk yes something along those lines would be ideal, have @BOARD_BUILD_DIR@ and maybe @BLOB_DIR@ as well
@SergiiDmytruk I like the @VAR@ expansion commit, but given the size of this PR, it might be better to do separately
@MrChromebox I am still unsure how to properly manage big changes under Heads like this one which adds a new arch and requires Makefile, modules and boards specific changes altogether.
How would you seperate this in smaller PRs?
@MrChromebox @tlaurion Changes fall into the following categories:
- changing build layout
- adding power arch
- adding new boards
- smaller changes like building musl in parallel
I can make a separate PR with non-Talos changes and this one will be rebased on top of it. Or two new PRs: build layout change in the first and everything else but new boards in the second.
@tlaurion I'm not seeing why they can't be broken down as @SergiiDmytruk mentioned above. I'd be ok with 3 PRs - one with the build changes / adding power arch, one for adding the new boards, and then the non-Talos changes as a third.