heads
heads copied to clipboard
linuxboot
Use nix for reproducible build env
Use nix for the build environment. This gives us a reproducible build environment. Use it by entering a "pure" nix shell so only what is specified in the shell.nix is available by running nix-shell --pure.
@tlaurion here's my branch.... finally! I ended up going with good old nix-shell w/ buildFHSUserEnv so we can have executables in /usr/bin/. I just gave it a go and it got much further than I've had so far but its stuck building coreboot-4.13/crossgcc/GCC:
| ^~
In file included from ../../gcc-8.3.0/gcc/backend.h:30,
from ../../gcc-8.3.0/gcc/lra-constraints.c:112:
../../gcc-8.3.0/gcc/bitmap.h:771:3: note: ...this statement, but the latter is misleadingly indented as if it were guarded by the 'if'
771 | for (bmp_iter_set_init (&(ITER), (BITMAP), (MIN), &(BITNUM)); \
| ^~~
../../gcc-8.3.0/gcc/lra-constraints.c:4579:5: note: in expansion of macro 'EXECUTE_IF_SET_IN_BITMAP'
4579 | EXECUTE_IF_SET_IN_BITMAP (&lra_reg_info[regno].insn_bitmap, 0, uid, bi)
| ^~~~~~~~~~~~~~~~~~~~~~~~
../../gcc-8.3.0/gcc/lto-compress.c:34:10: fatal error: zlib.h: No such file or directory
34 | #include <zlib.h>
| ^~~~~~~~
compilation terminated.
g++ -fno-PIE -c -O2 -fomit-frame-pointer -m64 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -pedantic -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -DHAVE_CONFIG_H -I. -I. -I../../gcc-8.3.0/gcc -I../../gcc-8.3.0/gcc/. -I../../gcc-8.3.0/gcc/../include -I../../gcc-8.3.0/gcc/../libcpp/include -I/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/xgcc/include -I/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/xgcc/include -I/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/xgcc/include -I../../gcc-8.3.0/gcc/../libdecnumber -I../../gcc-8.3.0/gcc/../libdecnumber/dpd -I../libdecnumber -I../../gcc-8.3.0/gcc/../libbacktrace -o omp-offload.o -MT omp-offload.o -MMD -MP -MF ./.deps/omp-offload.TPo ../../gcc-8.3.0/gcc/omp-offload.c
make[1]: *** [Makefile:1110: lto-compress.o] Error 1
make[1]: *** Waiting for unfinished jobs....
sh ../../gcc-8.3.0/gcc/../move-if-change tmp-attrtab.c insn-attrtab.c
sh ../../gcc-8.3.0/gcc/../move-if-change tmp-dfatab.c insn-dfatab.c
sh ../../gcc-8.3.0/gcc/../move-if-change tmp-latencytab.c insn-latencytab.c
echo timestamp > s-attrtab
sh ../../gcc-8.3.0/gcc/../move-if-change tmp-automata.c insn-automata.c
echo timestamp > s-automata
rm gcc.pod
make[1]: Leaving directory '/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/build-i386-elf-GCC/gcc'
make: *** [Makefile:4244: all-gcc] Error 2
You can give it a go by running nix-shell --pure and then make.
Ok made a little more progress, got past the zlib.h issue and added a few more package after trying from scratch with nix-shell --pure. But now we're running into build errors build GCC.
g++ -fno-PIE -c -I../../gcc-8.3.0/gcc/../libgcc -DEH_MECHANISM_gcc -DIN_GCC_FRONTEND -O2 -fomit-frame-pointer -m64 -DIN_GCC -DCROSS_DIRECTORY_STRUCTURE -fno-exceptions -fno-rtti -fasynchronous-unwind-tables -W -Wall -Wno-narrowing -Wwrite-strings -Wcast-qual -Wmissing-format-attribute -Woverloaded-virtual -Wno-long-long -Wno-variadic-macros -Wno-overlength-strings -DHAVE_CONFIG_H -I. -Iada -I../../gcc-8.3.0/gcc -I../../gcc-8.3.0/gcc/ada -I../../gcc-8.3.0/gcc/../include -I../../gcc-8.3.0/gcc/../libcpp/include -I/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/xgcc/include -I/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/xgcc/include -I/home/manny/projects/github.com/osresearch/heads/build/x86/coreboot-4.13/util/crossgcc/xgcc/include -I../../gcc-8.3.0/gcc/../libdecnumber -I../../gcc-8.3.0/gcc/../libdecnumber/dpd -I../libdecnumber -I../../gcc-8.3.0/gcc/../libbacktrace -o ada/raise-gcc.o -MT ada/raise-gcc.o -MMD -MP -MF ada/.deps/raise-gcc.TPo ../../gcc-8.3.0/gcc/ada/raise-gcc.c
In file included from /nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/signal.h:328,
from /nix/store/mmmjp0xcmr0dm307jv1l34hvi95z6d8s-glibc-2.35-224-dev/include/sys/param.h:28,
from ../../gcc-8.3.0/gcc/system.h:298,
from ../../gcc-8.3.0/gcc/ada/init.c:65:
../../gcc-8.3.0/gcc/ada/init.c:575:18: error: missing binary operator before token "("
575 | # if 16 * 1024 < MINSIGSTKSZ
| ^~~~~~~~~~~
Next step is to try with an older gcc instead, currently its at 11.3.
@mmlb : Issues are opened upstream (nixos) for GCC and Ada.
General idea (fixed for la test ada) seems to always try to bootstrap ADA from an older version to build newer versions. Haven't checked progression of issues but important as part of coreboot builder project as well
Edit
https://github.com/NixOS/nixpkgs/issues/174982
So I guess target would be gnat6 to obtain both older GCC and ada to build musl-cross-make and each coreboot's dependent version buildstack?
@mmlb : Issues are opened upstream (nixos) for GCC and Ada.
General idea (fixed for la test ada) seems to always try to bootstrap ADA from an older version to build newer versions. Haven't checked progression of issues but important as part of coreboot builder project as well
Edit
So I guess target would be gnat6 to obtain both older GCC and ada to build musl-cross-make and each coreboot's dependent version buildstack?
Yep I'm trying that now locally. Looks like its going to build gcc/gnat for the nixpkgs revision I pinned to. Now I remember why the initial flake had the old gnat, I had a hunch that newer gcc/gnat would fail to build old one and went back and found out I was right :D. I'll see if gnat builds now and also check around to see if there's a revision that has gnat built by hydra so users won't have to rebuild gnat/gcc.
Ok back to nix flakes. The nice thing about flakes is that its easy to manage the pinned versions as flakes locks the inputs and knows how to update them. Also makes it really easy to add new inputs of nixpkgs at specific commits for example.
So I've switched back over to the flake so that I can grab the last "known good" version of gnat6 (from an old nixpkgs commit), latest nixpkgs gnat6 still doesn't build for me. Unfortunately the old gnat6 + buildFHSUserEnv don't seem to play well because I'm now getting:
error while loading shared libraries: __vdso_gettimeofday: invalid mode for dlopen(): Invalid argument
I'm going to go back to nixpkgs and see whats up with gnat6 on master and if there's anything to be done about vdso.
--- edit
Also, the coreboot patch is so coreboot will accept gnatbind over just gnat. Not sure if thats alright or not, will have to see once vdso is issue is figured out.
@osresearch @mmlb Have you fell upon https://git.sr.ht/~amjoseph/ownerboot ? Fascinating food for thought.
@tlaurion I had not see that no, thanks. All of the successful things I've seen so far are building coreboot with nix and nix built toolchains, which is not the route I thought we wanted to be going initially. I think we might want to reconsider that, maybe we build coreboot with make but with nix's gcc as the initial stab instead?
@mmlb
Well, depending of how incremental we want changes to Heads to happen (Makefile)
- If gnat6 and GCC are provided in earlier versions enough to build coreboot 4.11, nix toolchain can be used to bootstrap coreboot per version buildstack, otherwise we won't be able to build kgpe-d16 and librem server (based on coreboot 4.11).
- As of now, everything else Heads uses musl-cross-make to build modules.
So to go the route of using coreboot buildstack under heads, nix would either have to also support older versions of coreboot (not the case right now, I think that 4.16+ are supported under nix).
Otherwise, we could switch things around and get rid of musl-cross-make and use coreboots buoldstack, but I'm worried that would increase maintainsership cost. What is interesting about ownerboot is that they are building initrd tools directly from nix buikdstack as well to be packed. This is neat, but we would need to use something stable as a buildstack there. That would be better then using per version coreboot buildstaxk to build everything inside of heads, which would also augment cahce differences we would need to keep since they won't be shareable between different board's coreboot version depending buildstack.
So my thoughts on previous comment of yours is that:
- best avenue would be to have earlier version of gnat and GCC to bootstrap coreboot's buildstack. That seems to be the best avenue, but would require upstream nixos work and bugfixes I guess.
@mmlb
Well, depending of how incremental we want changes to Heads to happen (Makefile)
* If gnat6 and GCC are provided in earlier versions enough to build coreboot 4.11, nix toolchain can be used to bootstrap coreboot per version buildstack, otherwise we won't be able to build kgpe-d16 and librem server (based on coreboot 4.11). * As of now, everything else Heads uses musl-cross-make to build modules.So to go the route of using coreboot buildstack under heads, nix would either have to also support older versions of coreboot (not the case right now, I think that 4.16+ are supported under nix).
I don't think I follow here. Are you suggesting to use coreboot's build of gcc stack for the rest of heads instead of just for coreboot? Thats not exactly what I was suggesting earlier. I was suggesting we use the same gcc setup for both coreboot and rest of heads. (I wasn't aware of usage of musl-cross-make, I assumed heads used the same compilers coreboot builds/uses. I'm not sure what it would take to replace musl-cross-make if wanted.
Otherwise, we could switch things around and get rid of musl-cross-make and use coreboots buoldstack, but I'm worried that would increase maintainsership cost. What is interesting about ownerboot is that they are building initrd tools directly from nix buikdstack as well to be packed. This is neat, but we would need to use something stable as a buildstack there. That would be better then using per version coreboot buildstaxk to build everything inside of heads, which would also augment cahce differences we would need to keep since they won't be shareable between different board's coreboot version depending buildstack.
Yep this is what I was suggesting too. You mention:
This is neat, but we would need to use something stable as a buildstack there.
What do you mean when you say "stable"? Why isn't nix's gcc+ada considered stable for this purpose?
So my thoughts on previous comment of yours is that:
* best avenue would be to have earlier version of gnat and GCC to bootstrap coreboot's buildstack. That seems to be the best avenue, but would require upstream nixos work and bugfixes I guess.
So I took a look at fixing gnat6 in latest nixpkgs instead of relying on an old one which should get past the gettimeofday vdso issue I think. This isn't currently working due to bootstrapping issues. nixpkgs uses binary builds of gnat from https://github.com/alire-project/GNAT-FSF-builds to build nixpkg's gcc+ada. Both nixpkgs and alire-project only actually go back to v11 :(. So this needs to be figured out somehow :thinking:.
don't think I follow here. Are you suggesting to use coreboot's build of gcc stack for the rest of heads instead of just for coreboot? Thats not exactly what I was suggesting earlier. I was suggesting we use the same gcc setup for both coreboot and rest of heads. (I wasn't aware of usage of musl-cross-make, I assumed heads used the same compilers coreboot builds/uses. I'm not sure what it would take to replace musl-cross-make if wanted.
As of now:
- host GCC and gnat under debian-10-11 are used to compile musl-cross-make and each versions of compiled board defined coreboot version's buildstack
- musl-cross-make is used to build all modules/* defined under board config and their dependencies outsider of coreboot
- coreboot is finally built using board's defined coreboot's version buildstack
So following your comment:
- musl-cross-make could probably be replaced by nix's reproducible toolstack, but that would require changes to heads Make file to start transition.
I thought the first step would be to have nix have an earlier version if the toolstack (mostly for gnat) to be able to compile coreboot 4.11 and later buildstack AND musl-cross-make.
If we can also use Nix's musl-cross-make builds: awesome. There will just need to have some adaptation to Heads makefile, since first thing built is musl-cross-make here.
@mmlb makes sense?
So I took a look at fixing gnat6 in latest nixpkgs instead of relying on an old one which should get past the gettimeofday vdso issue I think. This isn't currently working due to bootstrapping issues. nixpkgs uses binary builds of gnat from https://github.com/alire-project/GNAT-FSF-builds to build nixpkg's gcc+ada. Both nixpkgs and alire-project only actually go back to v11 :(. So this needs to be figured out somehow thinking.
@mmlb any news? :/
@tlaurion nope sorry I got pulled off into other things at work and at home. I was actually thinking about this earlier today. I tried out https://github.com/alire-project/GNAT-FSF-builds on my own but got no where. I think its way past time I solicit the help of others. I'll ask around in nix's matrix to see how we can get gnat6.
[12:45:20]-[~/c/g/N/nixpkgs]-[manny@c3mnix]
nix-build -A gnatboot4 -o gnatboot4; and echo gnatboot4:; and ./gnatboot4/bin/gcc --version; and nix-build -A gnat6 -o gnat6; and echo gnat6:; and ./gnat6/bin/gnat -v
/nix/store/f1f7zij1f8njddwcgig9idh0mwxg8211-gnatboot-wrapper-4.1
gnatboot4:
gcc (GCC) 4.1.0
Copyright (C) 2006 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
/nix/store/iz40d67dkvyfk4yd0pz5kkfyv8p0fqq6-gnat-wrapper-6.5.0
gnat6:
GNAT 6.5.0
Copyright 1996-2016, Free Software Foundation, Inc.
List of available commands
gnat bind gnatbind
gnat chop gnatchop
gnat clean gnatclean
gnat compile gnatmake -f -u -c
gnat check gnatcheck
gnat elim gnatelim
gnat find gnatfind
gnat krunch gnatkr
gnat link gnatlink
gnat list gnatls
gnat make gnatmake
gnat metric gnatmetric
gnat name gnatname
gnat preprocess gnatprep
gnat pretty gnatpp
gnat stack gnatstack
gnat stub gnatstub
gnat test gnattest
gnat xref gnatxref
Commands bind, find, link, list and xref accept project file switches -vPx, -Pprj, -Xnam=val,--subdirs= and -eL
:tada:
Now to open up a PR to nixpkgs.
@mmlb
In parallel, trying to make boards not depend on ada by removing libgfx support on all boards, explicitly relying on board config's kernel configs to provide fb through drm that will either work by abusing kernel fb exposition (no compression of fb, exposing fb address new config options that taint Heads on 5.x kernel for i915 driver), hacking kexec to provide exposed flat fb address in kexec call (whitelisting approch under kexec patch in master) or require that the next kexec'ed kernel provides DRM+GPU in initrd so there is fb avail when asking for LUKS decryption key passphrase.
First results shows that most boards did not really need coreboot libgfx for all non-dgpu enabled boards. (See recent master changes for librem boards).
If tests with dgpu vbios provided blobs also confirms non requirement of libgfx (they shouldn't, only doubt is for EDP variant of x230 which provides modified vbt which seems to be relaying on libgfx at least in doc, but a quick test by board owners should validate/invalidate this.) Then ada requirement to libgfxiniy will be gone for Heads.
Long story short, we might not need ada altogether. History will tell soon enough.
Crossposting here from https://github.com/NixOS/nixpkgs/pull/225191#issuecomment-1537177144
@mmlb well some progress, but as of 6423000 on CircleCI:
https://app.circleci.com/pipelines/github/tlaurion/heads/1683/workflows/97b36967-28ba-47be-8fe9-24714bd81ab1/jobs/22773/parallel-runs/0/steps/0-102
Builds ends with
** Message: 17:51:51.400: Requires Linux version >= 3.19 built with CONFIG_USER_NS ** Message: 17:51:51.402: Run: sudo sysctl -w kernel.unprivileged_userns_clone=1
** (process:582): ERROR **: 17:51:51.402: main: unshare: Operation not permitted
Exited with code exit status 133
@tlaurion I took your branch and put my nix bits on top (https://github.com/mmlb/osresearch-heads/tree/stage_all_nixos_local_buildsystem_flake) (killed your nix stuff for now since it was making direnv hang for me), but its still looking for gnat :(. I'm testing with just make -j1 have something else I can run to test?
crossgcc/build-i386-elf-GCC/build.log
Ahh if I get rid of my gnat6 then crossgcc just throws a scary warning but continues, I've been stopping it before it continues :smile:.
Are you on matrix? On heads channel? Just invited a mmlb on the channel.
#OSFW-Heads:matrix.org
@mmlb seems like this explains why nix-shell pure has problems with downloading gentoo without exported CA on your branch nix'CI tests? https://github.com/mmlb/osresearch-heads/tree/stage_all_nixos_local_buildsystem_flake
https://review.coreboot.org/c/coreboot/+/60065
Testing against staging PR https://github.com/osresearch/heads/compare/master...tlaurion:heads:staging_all-gpg240-tcpa_rework-nix
Working on other things, we realised that sysroot was not passed from global Makefile, which made some autotools not look for libs/ headers at the right place and misbehave. Maybe take a look at the global Makefile? Other eyes there would be welcome.
Not sure it will fix the failing builds here but it might.
Building at https://app.circleci.com/pipelines/github/tlaurion/heads/1790/workflows/cc133870-8895-481b-b5f4-1e2730dc0ad1/jobs/26021/parallel-runs/0/steps/0-102
@mmlb nope, library still not found at linking.... https://app.circleci.com/pipelines/github/tlaurion/heads/1790/workflows/cc133870-8895-481b-b5f4-1e2730dc0ad1/jobs/26022/parallel-runs/0/steps/0-102
@tlaurion, I'm trying to grasp the task of reproducible builds here and how it could be performed. Please take a look.
My understanding of the situation
So the idea is to use Nix as a predictable build environment as part of reproducible-builds setup that can also be employed on CI. Nix is to be used in a "pure" way as much as possible to avoid introducing accidental dependencies from the build machine. Guix was discarded because of some dependencies missing. Support of multiple (some old) coreboot versions is required.
Main issues
Nix requires absence of absolute paths, so need to get rid of those. Might also have to bump something to make it build as a last resort.
The issue with old gnat.
Challenges
Nix is unlikely to ever provide support for the needed coreboot toolchains, so they should be built by Heads after making Nix capable of building them.
Having a "pure" environment might not actually be easy to achieve without using a container. Even nix-shell --pure leaves $HOME, so shell's configuration might get picked up, flakes seems to have a similar issue. This might not pose a serious problem for the build and can probably be worked around.
Questions
Is this the minimal goal here?
- Make Heads actually finish its build under Nix (one board at first, then all the rest).
- Integrate gnat fix (given that upstream doesn't seem to want it much, can probably be built locally).
- Make Nix-based build work on CI with the necessary caching for adequate speed.
- Document how to perform the build.
Is using linux-builder part of this? It seem to be the next step after Nix is in place, but I'm not sure when (if ever) it should replace Makefiles.
The issue with old gnat.
I'm not fully up to date on this PR, but the old gnat problem was solved in https://github.com/linuxboot/heads/commit/e3805392027e2834a0db4dc9ab3478214ab6b7d8 by disabling it. Neither of the boards that are stuck on 4.11 actually needs gnat (they do not use libgfxinit).
I'm not fully up to date on this PR, but the old gnat problem was solved in e380539 by disabling it. Neither of the boards that are stuck on 4.11 actually needs gnat (they do not use libgfxinit).
Thanks, @JonathonHall-Purism. I know it was talked about, but didn't see that it has landed. This should make things easier.
Hey. IIRC the last issue that blocked me here was something wrt pkg-config. That was a few months ago though so probably should be checked out again. I'll push up the latest tree I have that I just rebased on master.
I've looked back at this a little, and the last commit was attempting to rework parameters passed from Makefile to build system and modules built. I rebased as well yersterday and retrying it from what would be equivalent of https://github.com/linuxboot/heads/pull/1269/commits/dc3ebbc67309075143334709d4313d82a63b6116
I'm continuing at https://github.com/tlaurion/heads/tree/nix_buildstack-mmlb-equiv_rebased_master-rework
Hey. IIRC the last issue that blocked me here was something wrt pkg-config. That was a few months ago though so probably should be checked out again. I'll push up the latest tree I have that I just rebased on master.
Yes. This is related to changes in Makefile. With pkg-config parametrs missing, the buildsystem was looking for linking in host system not under install dir for dependencies. Will update when I get an understanding of what is missing and what is the state of this PR. The goal here is to make sure that the global Makefile pass everything needed for modules configure/ compilation not bleeding/looking/escaping to host exposed tools.
Having a "pure" environment might not actually be easy to achieve without using a container. Even nix-shell --pure leaves $HOME, so shell's configuration might get picked up, flakes seems to have a similar issue. This might not pose a serious problem for the build and can probably be worked around.
From my understanding, what @mmlb tried here (OP is misleading since he got away of a "pure" nix environment really short after starting this works) by now using:
- nix container for build https://github.com/linuxboot/heads/compare/master...tlaurion:heads:nix_buildstack-mmlb-equiv_rebased_master-rework#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R145
- flake + develop nix environement https://github.com/linuxboot/heads/compare/master...tlaurion:heads:nix_buildstack-mmlb-equiv_rebased_master-rework#diff-78a8a19706dbd2a4425dd72bdab0502ed7a2cef16365ab7030a5a0588927bf47R17
We make sure that flake.lock and flake dependencies are the only ones exposed in the building environment.
I haven't been able to produce a proper Docker image from Dockerfile as of now to reuse, but I think the state of this PR is not ready to do that and a lot is still missing here. CircleCI uses image: nixos/nix:2.16.1 and then apply flakes pinned dependencies, versioned or not as of now. flake.lock "pins" the nixpkgs to a package source list, preventing to use newer package from nix upstream and permitting flakes.nix to specify a package name only, where used version is fixated by the pinned package list from flake.lock
That's where i'm at right now, trying to recreate local environment to reassess state of this project and then answer to your other questions @SergiiDmytruk, while trying to have CircleCI succeed builds.
EDIT:
TLDR: https://github.com/linuxboot/heads/pull/1269/commits/2f9b6b332f8018d4393739306eb4a796104a77bd was added by error prior of out of band discussions with mmlb. It was a foolish atempt of mine to fix cairo, and broke way more things. Restarting at https://github.com/linuxboot/heads/pull/1269/commits/dc3ebbc67309075143334709d4313d82a63b6116 shows not so much is missing to have PoC building all boards at https://app.circleci.com/pipelines/github/tlaurion/heads/2232/workflows/4c9f33a2-ead4-4b07-96d9-d148ba755ed7
Notes:
- prep_step still use debian-11. This is because even on debian-12, gems used by vbios download/extract scripts are not working, it is implied the vbios download scripts will need simplification prior of being able to work without hassle.
- All other build steps (having blobs in place, where everything not vbios currently works and does not depend on manual invocation of scripts) from CircleCI prior of launching the builds. The only reason the blobs download/extraction is made in prep_step is to download once, do a workspace cache and pass that workspace cache to all following buildsteps to not redownload constantly from manufacturers websites
OP is misleading since he got away of a "pure" nix environement since starting
There is --ignore-environment in there for nix develop which is kind of like nix shell --pure.
I haven't been able to produce a proper Docker image from Dockerfile as of now to reuse,
docker build -t heads-nix . worked without issues for me, but I have Nix installed anyway, so just using nix develop -i.