heads
heads copied to clipboard
linuxboot
Review cross compilation best practices effectively enforced in global Makefile and submodules
Trying to build Heads latest commit for the board ** results in:
$ git log --oneline --no-decorate -1
59972f3 Merge pull request #1459 from JonathonHall-Purism/hires_scale
$ make BOARD_NAME=qemu-coreboot-whiptail-tpm2-hotp
[…]
2023-08-12 11:00:07+02:00 WGET https://www.gnupg.org/ftp/gcrypt/gnupg/gnupg-2.4.0.tar.bz2
--2023-08-12 11:00:07-- https://www.gnupg.org/ftp/gcrypt/gnupg/gnupg-2.4.0.tar.bz2
Resolving www.gnupg.org... 217.69.76.60, 2001:aa8:fff1:2100::60
Connecting to www.gnupg.org|217.69.76.60|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7666935 (7.3M) [text/plain]
Saving to: ‘/dev/shm/heads/packages/x86/gnupg-2.4.0.tar.bz2.tmp’
/dev/shm/heads/packages/x86/gnupg-2.4 100%[=======================================================================>] 7.31M 44.5MB/s in 0.2s
2023-08-12 11:00:07 (44.5 MB/s) - ‘/dev/shm/heads/packages/x86/gnupg-2.4.0.tar.bz2.tmp’ saved [7666935/7666935]
echo "1d79158dd01d992431dd2e3facb89fdac97127f89784ea2cb610c600fb0c1483 /dev/shm/heads/packages/x86/gnupg-2.4.0.tar.bz2" | sha256sum --check -
/dev/shm/heads/packages/x86/gnupg-2.4.0.tar.bz2: OK
mkdir -p "/dev/shm/heads/build/x86/gnupg-2.4.0/"
tar -xf "/dev/shm/heads/packages/x86/gnupg-2.4.0.tar.bz2" --strip 1 -C "/dev/shm/heads/build/x86/gnupg-2.4.0/"
if [ -r patches/gpg2-2.4.0.patch ]; then ( git apply --verbose --reject --binary --directory build/x86/gnupg-2.4.0 ) < patches/gpg2-2.4.0.patch || exit 1 ; fi
Checking patch build/x86/gnupg-2.4.0/configure...
Checking patch build/x86/gnupg-2.4.0/common/ttyio.c...
Applied patch build/x86/gnupg-2.4.0/configure cleanly.
Applied patch build/x86/gnupg-2.4.0/common/ttyio.c cleanly.
if [ -d patches/gpg2-2.4.0 ] && [ -r patches/gpg2-2.4.0 ] ; then for patch in patches/gpg2-2.4.0/*.patch ; do echo "Applying patch file : $patch " ; ( git apply --verbose --reject --binary --directory build/x86/gnupg-2.4.0 ) < $patch || exit 1 ; done ; fi
2023-08-12 11:00:08+02:00 CONFIG gpg2
make: *** [Makefile:461: /dev/shm/heads/build/x86/gnupg-2.4.0/.configured] Error 1
From build/x86/gnupg-2.4.0/config.log:
It was created by gnupg configure 2.4.0, which was
generated by GNU Autoconf 2.69. Invocation command line was
$ ./configure CPPFLAGS=-I/dev/shm/heads/install/x86/include/libusb-1.0 --host x86_64-linux-musl --with-libusb=/dev/shm/heads/install/x86 --with-gpg-error-prefix=/dev/shm/heads/install/x86 --with-libgcrypt-prefix=/dev/shm/heads/install/x86 --with-libassuan-prefix=/dev/shm/heads/install/x86 --with-ksba-prefix=/dev/shm/heads/install/x86 --with-npth-prefix=/dev/shm/heads/install/x86 --prefix / --libexecdir /bin --enable-scdaemon --enable-ccid-driver --disable-tofu --disable-rpath --disable-regex --disable-doc --disable-bzip2 --disable-exec --disable-photo-viewers --disable-ldap --disable-regex --disable-nls --disable-all-tests --disable-wks-tools --disable-gnutls --disable-dirmngr --disable-ntbtls --disable-libdns --disable-zip --disable-sqlite --disable-gpgsm
configure:16564:
***
*** You need libgpg-error to build this program.
** This library is for example available at
*** https://gnupg.org/ftp/gcrypt/gpgrt
*** (at least version 1.46 is required.)
***
configure:16615:
***
*** You need libksba to build this program.
*** This library is for example available at
*** https://gnupg.org/ftp/gcrypt/libksba/
*** (at least version 1.6.3 using API 1 is required).
***
configure:16681: error:
***
*** Required libraries not found. Please consult the above messages
*** and install them before running configure again.
***
libgpg-error and libksba should be there:
$ ls install/x86/
bin include lib man sbin share
$ fd -I gpg-error install/x86
install/x86/bin/gpg-error
install/x86/bin/gpg-error-config
install/x86/include/gpg-error.h
install/x86/lib/libgpg-error.la
install/x86/lib/libgpg-error.so
install/x86/lib/libgpg-error.so.0
install/x86/lib/libgpg-error.so.0.33.1
install/x86/lib/pkgconfig/gpg-error.pc
install/x86/share/aclocal/gpg-error.m4
$ fd -I ksba install/x86
install/x86/bin/ksba-config
install/x86/include/ksba.h
install/x86/lib/libksba.la
install/x86/lib/libksba.so
install/x86/lib/libksba.so.8
install/x86/lib/libksba.so.8.14.3
install/x86/lib/pkgconfig/ksba.pc
install/x86/share/aclocal/ksba.m4
install/x86/share/info/ksba.info
libgpg-error and libksba should be there:
$ ls install/x86/ bin include lib man sbin share $ fd -I gpg-error install/x86 install/x86/bin/gpg-error install/x86/bin/gpg-error-config install/x86/include/gpg-error.h install/x86/lib/libgpg-error.la install/x86/lib/libgpg-error.so install/x86/lib/libgpg-error.so.0 install/x86/lib/libgpg-error.so.0.33.1 install/x86/lib/pkgconfig/gpg-error.pc install/x86/share/aclocal/gpg-error.m4 $ fd -I ksba install/x86 install/x86/bin/ksba-config install/x86/include/ksba.h install/x86/lib/libksba.la install/x86/lib/libksba.so install/x86/lib/libksba.so.8 install/x86/lib/libksba.so.8.14.3 install/x86/lib/pkgconfig/ksba.pc install/x86/share/aclocal/ksba.m4 install/x86/share/info/ksba.info
@paulmenzel that is actually the issue: it's already there.
I would suggest building with make BOARD=xyz real.clean and try again. What is under install dir is searched for when pkg-config is called and finds older versions. Those errors do not happen when building clean. The whole Makefile logic won't reapply when build/arch/package/.* are present and won't delete older libs under install/arch/.
The modules are not enough intelligent to clean past builds artifacts lying around under install dir.
You could hack around removing those specific libs under install, respecting modules dependency graph and be successful; I did that myself when bumping versions before.
libgpg-error and libksba should be there:
$ ls install/x86/ bin include lib man sbin share $ fd -I gpg-error install/x86 install/x86/bin/gpg-error install/x86/bin/gpg-error-config install/x86/include/gpg-error.h install/x86/lib/libgpg-error.la install/x86/lib/libgpg-error.so install/x86/lib/libgpg-error.so.0 install/x86/lib/libgpg-error.so.0.33.1 install/x86/lib/pkgconfig/gpg-error.pc install/x86/share/aclocal/gpg-error.m4 $ fd -I ksba install/x86 install/x86/bin/ksba-config install/x86/include/ksba.h install/x86/lib/libksba.la install/x86/lib/libksba.so install/x86/lib/libksba.so.8 install/x86/lib/libksba.so.8.14.3 install/x86/lib/pkgconfig/ksba.pc install/x86/share/aclocal/ksba.m4 install/x86/share/info/ksba.info
@paulmenzel that is actually the issue: it's already there.
I would suggest building with make BOARD=xyz real.clean and try again. What is under install dir is searched for when pkg-config is called and finds older versions. Those errors do not happen when building clean. The whole Makefile logic won't reapply when build/arch/package/.* are present and won't delete older libs under install/arch/.
The modules are not enough intelligent to clean past builds artifacts lying around under install dir.
You could hack around removing those specific libs under install, respecting modules dependency graph and be successful; I did that myself when bumping versions before.
Also you should use https://github.com/osresearch/heads/pull/1403 for qemu boards
I would suggest building with make BOARD=xyz real.clean and try again.
No, I still get the same problem:
$ make BOARD=qemu-coreboot-fbwhiptail-tpm2-hotp real.clean
[…]
$ make BOARD=qemu-coreboot-fbwhiptail-tpm2-hotp
[…]
2023-08-13 23:32:29+02:00 CONFIG gpg2
make: *** [Makefile:461: /dev/shm/heads/build/x86/gnupg-2.4.0/.configured] Error 1
$ less build/x86/gnupg-2.4.0/config.log
[…]
configure:16564:
***
*** You need libgpg-error to build this program.
** This library is for example available at
*** https://gnupg.org/ftp/gcrypt/gpgrt
*** (at least version 1.46 is required.)
***
configure:16615:
***
*** You need libksba to build this program.
*** This library is for example available at
*** https://gnupg.org/ftp/gcrypt/libksba/
*** (at least version 1.6.3 using API 1 is required).
***
configure:16681: error:
***
*** Required libraries not found. Please consult the above messages
*** and install them before running configure again.
***
[…]
@paulmenzel I'm confused since debian-11, used per CircleCI, is building all boards without cache successfully for each commit (including gpg2 since version bump was merged), where i'm building locally on top of debian-12 without issues as well.
Can you maybe drop the whole configure.logs?
My hypothesis is that pkg-config is misbehaving here and for some reason looking for host's pkg-config instead of what is found under /install, which is fragile and varies between host's buildstacks which varies from one linux distribution to another.
Looking for the day where #1269 will be accomplished and we pin shell-env to desired and isolated host exposed minimal toolstack to not have those kind of issues ever again for a specific Heads commit forever (and where if we have proper mirrors used to point tarballs download across the whole ecosystem, we might finally have reproducible builds forever as well). But we are not there yet, and meanwhile, the only recommended buildstack is what is reproduced from CircleCI and that is debian-11 with what is expected toolstack installed in the host is exposed at apt calls in the CircleCI configuration.
@paulmenzel I'm sorry, but there is no path exposed in there that are useful....
Maybe make -d BOARD=xyz V=1? > log.txt
On debian-11? You didn't specify OS yet. Again I repeat, cannot replicate on debian-11/12 :/
You can inspect logs artifacts of the qemu-coreboot-fbwhiptail-tpm1 for comparison from CircleCI checks of each commit
$ make -d BOARD=qemu-coreboot-fbwhiptail-tpm2-hotp V=1 |& tee /scratch/tmp/20230816-heads-qemu-coreboot-fbwhiptail-tpm2-hotp.log
Rename to remove the .gz suffix necessary to upload to GitHub: 20230816-heads-qemu-coreboot-fbwhiptail-tpm2-hotp.log.7z.gz
(342 MiB compressed to 11 MB with 7z a -t7z -m0=lzma -mx=9 -mfb=64 -md=32m -ms=on /scratch/tmp/20230816-heads-qemu-coreboot-fbwhiptail-tpm2-hotp.log.7z /scratch/tmp/20230816-heads-qemu-coreboot-fbwhiptail-tpm2-hotp.log.)
On debian-11? You didn't specify OS yet. Again I repeat, cannot replicate on debian-11/12 :/
I am using MarIuX, a distribution once based on Linux From Scratch.
Ok, config.log contains
[…]
configure: checking for libraries
checking for gpg-error-config... /dev/shm/heads/install/x86/bin/gpg-error-config
checking for gpgrt-config... /usr/bin/gpgrt-config
configure: Use gpgrt-config with /usr/lib as gpg-error-config
checking for GPG Error - version >= 1.46... no
configure: Use gpgrt-config as libgcrypt-config
checking for LIBGCRYPT - version >= 1.9.1... yes (1.9.4)
checking LIBGCRYPT API version... okay
configure: WARNING:
***
*** The config script "/usr/bin/gpgrt-config --libdir=/usr/lib libgcrypt" was
*** built for x86_64-pc-linux-gnu and thus may not match the
*** used host x86_64-pc-linux-musl.
*** You may want to use the configure option --with-libgcrypt-prefix
*** to specify a matching config script or use $SYSROOT.
***
configure: Use gpgrt-config as libassuan-config
checking for LIBASSUAN - version >= 2.5.0... yes (2.5.3)
checking LIBASSUAN API version... okay
configure: WARNING:
***
*** The config script "/usr/bin/gpgrt-config --libdir=/usr/lib libassuan" was
*** built for x86_64-pc-linux-gnu and thus may not match the
*** used host x86_64-pc-linux-musl.
*** You may want to use the configure option --with-libassuan-prefix
*** to specify a matching config script.
***
configure: Use gpgrt-config as ksba-config
checking for KSBA - version >= 1.6.3... no
[…]
and adding /dev/shm/heads/install/x86/bin to the path, fixes the problem.
$ type gpgrt-config
gpgrt-config is /usr/bin/gpgrt-config
$ export PATH="/dev/shm/heads/install/x86/bin:$PATH"
$ type gpgrt-config
gpgrt-config is /dev/shm/heads/install/x86/bin/gpgrt-config
$ CC="/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-gcc -fdebug-prefix-map=/dev/shm/heads=heads -gno-record-gcc-switches -D__MUSL__ -isystem /dev/shm/heads/install/x86/include -L/dev/shm/heads/install/x86/lib" AR=/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-ar LD=/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-ld STRIP=/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-strip NM=/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-nm OBJCOPY=/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-objcopy OBJDUMP=/dev/shm/heads/crossgcc/x86/bin/x86_64-linux-musl-objdump PKG_CONFIG_PATH= PKG_CONFIG_LIBDIR=/dev/shm/heads/install/x86/lib/pkgconfig PKG_CONFIG_SYSROOT_DIR=/dev/shm/heads/install/x86 CFLAGS=-Os ./configure CPPFLAGS=-I/dev/shm/heads/install/x86/include/libusb-1.0 --host x86_64-linux-musl --with-libusb=/dev/shm/heads/install/x86 --with-gpg-error-prefix=/dev/shm/heads/install/x86 --with-libgcrypt-prefix=/dev/shm/heads/install/x86 --with-libassuan-prefix=/dev/shm/heads/install/x86 --with-ksba-prefix=/dev/shm/heads/install/x86 --with-npth-prefix=/dev/shm/heads/install/x86 --prefix / --libexecdir /bin --enable-scdaemon --enable-ccid-driver --disable-tofu --disable-rpath --disable-regex --disable-doc --disable-bzip2 --disable-exec --disable-photo-viewers --disable-ldap --disable-regex --disable-nls --disable-all-tests --disable-wks-tools --disable-gnutls --disable-dirmngr --disable-ntbtls --disable-libdns --disable-zip --disable-sqlite --disable-gpgsm
[…]
checking for gpg-error-config... /dev/shm/heads/install/x86/bin/gpg-error-config
checking for gpgrt-config... /dev/shm/heads/install/x86/bin/gpgrt-config
[…]
configure: WARNING: unrecognized options: --with-libusb, --disable-regex, --disable-regex
GnuPG v2.4.0 has been configured as follows:
Revision: c0556edb8 (49237)
Platform: GNU/Linux (x86_64-pc-linux-musl)
OpenPGP: yes
S/MIME: no
Agent: yes
Smartcard: yes
TPM: no
G13: no
Dirmngr: no
Keyboxd: no
Gpgtar: yes
WKS tools: no
Protect tool: (default)
LDAP wrapper: (default)
Default agent: (default)
Default pinentry: (default)
Default scdaemon: (default)
Default keyboxd: (default)
Default tpm2daemon: (default)
Default dirmngr: (default)
Dirmngr auto start: yes
Readline support: no
LDAP support: no
TLS support: no
TOFU support: no
Tor support: only .onion
$ make PATH="/dev/shm/heads/install/x86/bin:$PATH" BOARD=qemu-coreboot-fbwhiptail-tpm2-hotp
works around the error (and hits a new one, reported in issue https://github.com/osresearch/heads/issues/1470).
@paulmenzel : This is relevant for #1269. Not sure why gpgrt-config is not picked up... Something missing into our SYSROOT under global Makefile I guess https://dev.gnupg.org/T3950 which was started to be applied in my attempt to help #1269 but is definitely not ready, but a base to make the buildsystem play more nicely with pkg-config and sysroot. Why it affects your distro but not debian? I would love to know.
Relative sysroot attempts https://github.com/osresearch/heads/commit/8e7952a0ff97e0f363fd19ae21256ca98e7e7d62
This issue is linked to host system paths bleeding into the build. Attempt to manage it globally with syroot would be needed since:
--with-libgcrypt-prefix --with-libassuan-prefix
Is not respected.
Also review configure options. The following are deprecated: --with-libusb, --disable-regex, --disable-regex
Host build environment leaks into builds.
Mitigation for dirmngr spawning even if explicitely asked to be configured deactivated at https://github.com/linuxboot/heads/pull/1580
Traces of what was attempted before https://github.com/linuxboot/heads/pull/1580#issuecomment-1883548724