heads icon indicating copy to clipboard operation
heads copied to clipboard

Published 20 hours ago • verified publisher icon linuxboot

busybox build does not reproduce

Open JonathonHall-Purism opened this issue 1 year ago • 1 comments

I compared a CI build of #1630 with my local build, the only remaining reproducibility issue is busybox. All coreboot stages now reproduce, payload differs in the initrd only.

diffoscope identifies busybox. There are a lot of differences in disassembly and relocations but I think they all stem from a difference in string data (which causes more layout differences, etc.):

│ │ │ ├── strings --all --bytes=8 {}
│ │ │ │ @@ -665,15 +665,16 @@
│ │ │ │  []A\A]A^A_
│ │ │ │  h[]A\A]A^A_
│ │ │ │  []A\A]A^A_
│ │ │ │  []A\A]A^A_
│ │ │ │  [A\A]A^A_]
│ │ │ │  AWAVAUATUH
│ │ │ │  []A\A]A^A_
│ │ │ │ -%Y-%m-%d %H:%M:%
│ │ │ │ +   @@@%Y-%m-%d %H:%M:%
│ │ │ │ +NO NAME    
│ │ │ │  JFS1BSFXI
│ │ │ │  hsqsreebb
│ │ │ │   multi-call binary.
│ │ │ │  No help available
│ │ │ │  section header
│ │ │ │  unknown user/group
│ │ │ │  keyword outside section
│ │ │ │ @@ -1969,15 +1970,14 @@
│ │ │ │  can't make FAT32 with >128 sectors/cluster
│ │ │ │  Device '%s':
│ │ │ │  heads:%u, sectors/track:%u, bytes/sector:%u
│ │ │ │  media descriptor:%02x
│ │ │ │  total sectors:%lu, clusters:%u, sectors/cluster:%u
│ │ │ │  FATs:2, sectors/FAT:%u
│ │ │ │  volumeID:%08x, label:'%s'
│ │ │ │ -NO NAME    
│ │ │ │  --More-- 
│ │ │ │  (%u%% of %lu bytes)
│ │ │ │  (Enter:next line Space:next page Q:quit R:show the rest)
│ │ │ │  mount('%s','%s','%s',0x%08lx,'%s'):%d
│ │ │ │  would do mount('%s','%s','%s',0x%08lx,'%s')
│ │ │ │  %s is write-protected, mounting read-only
│ │ │ │  unc=\\%s\%.*s,prefixpath=%s

Some date format string changed slightly, now starts with @@@. NO NAME also moved.

(left is CI, right is my local build, and I did a clean build by deleting build/x86 and install/x86 for this)

JonathonHall-Purism avatar Apr 09 '24 19:04 JonathonHall-Purism

@JonathonHall-Purism some top of the head insight.

modukes/busybox is actually the only one being rebuilt without caring of its .build status from global Makefile rules, bypassing them.

I wonder if this is not the root of the problem, considering that CI rebuilds it multiple times depending of where that board is in the chain of workspace cache transfers.

I think a little rework on modules/busybox (oldest Heads module?) refreshed and respecting proper build recipes could probably resolve the problem altogether with a little bit of investigation. Some timestamps get into that module build. I think that if both those issues are tackled, that module would be reproducible and we would get rid of:

2024-04-19 19:27:49-04:00 INSTALL-MODULE drivers/usb/storage/usb-storage.ko
/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-strip --preserve-dates --strip-debug -o "/tmp/tmp.PXsyEBOSV2/lib/modules/usb-storage.ko" "/home/user/heads/build/x86/linux-5.10.5/linux-qemu/drivers/usb/storage/usb-storage.ko" 
2024-04-19 19:27:49-04:00 SYMLINK bin/busybox
make -C /home/user/heads/build/x86/busybox-1.36.1 CC="/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-gcc -fdebug-prefix-map=/home/user/heads=heads -gno-record-gcc-switches -D__MUSL__ -isystem /home/user/heads/install/x86/include -L/home/user/heads/install/x86/lib " CROSS_COMPILE="/home/user/heads/crossgcc/x86/bin/x86_64-linux-musl-" CONFIG_PREFIX="/tmp/tmp.8WKh1F5vTB/bin/.." install | tee -a /home/user/heads/build/x86/log/busybox.log  
make[1]: Entering directory '/home/user/heads/build/x86/busybox-1.36.1'
rm -f .kernelrelease
echo 1.36.1 > .kernelrelease
/home/user/heads/build/x86/busybox-1.36.1/scripts/gen_build_files.sh /home/user/heads/build/x86/busybox-1.36.1 /home/user/heads/build/x86/busybox-1.36.1
make -f scripts/Makefile.build obj=scripts/basic
make -f scripts/Makefile.build obj=applets
make -f scripts/Makefile.build obj=.
make -f scripts/Makefile.build obj=scripts
make -f scripts/Makefile.build obj=applets
make -f scripts/Makefile.build obj=archival
make -f scripts/Makefile.build obj=archival/libarchive
make -f scripts/Makefile.build obj=console-tools
make -f scripts/Makefile.build obj=coreutils
make -f scripts/Makefile.build obj=coreutils/libcoreutils
make -f scripts/Makefile.build obj=debianutils
make -f scripts/Makefile.build obj=klibc-utils
make -f scripts/Makefile.build obj=e2fsprogs
make -f scripts/Makefile.build obj=editors
make -f scripts/Makefile.build obj=findutils
make -f scripts/Makefile.build obj=init
make -f scripts/Makefile.build obj=libbb
make -f scripts/Makefile.build obj=libpwdgrp
make -f scripts/Makefile.build obj=loginutils
make -f scripts/Makefile.build obj=mailutils
make -f scripts/Makefile.build obj=miscutils
make -f scripts/Makefile.build obj=modutils
make -f scripts/Makefile.build obj=networking
make -f scripts/Makefile.build obj=networking/libiproute
make -f scripts/Makefile.build obj=networking/udhcp
make -f scripts/Makefile.build obj=printutils
make -f scripts/Makefile.build obj=procps
make -f scripts/Makefile.build obj=runit
make -f scripts/Makefile.build obj=selinux
make -f scripts/Makefile.build obj=shell
make -f scripts/Makefile.build obj=sysklogd
make -f scripts/Makefile.build obj=util-linux
make -f scripts/Makefile.build obj=util-linux/volume_id
rm -f .old_version
DO_INSTALL_LIBS="" \
	/bin/sh /home/user/heads/build/x86/busybox-1.36.1/applets/install.sh /tmp/tmp.8WKh1F5vTB/bin/.. --symlinks
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/[ -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/[[ -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/arch -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ascii -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ash -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/awk -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/base32 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/basename -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/bunzip2 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/bzcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/bzip2 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/chattr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/chmod -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/clear -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cmp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cpio -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/crc32 -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cttyhack -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/cut -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/date -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/df -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/diff -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dirname -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/dmesg -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/du -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/echo -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/env -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/expr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/factor -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/fallocate -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/false -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/find -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/fold -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/getopt -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/grep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/groups -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/gunzip -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/gzip -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/head -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hexdump -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hexedit -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/hostid -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/id -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/install -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/kill -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/killall -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/less -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/link -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ln -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ls -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsattr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsof -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsscsi -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lsusb -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lzcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/lzma -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/md5sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mkdir -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mkfifo -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mknod -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mktemp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/more -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mount -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/mv -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nl -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nproc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/nslookup -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/paste -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/patch -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pgrep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pidof -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ping -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pkill -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/printf -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ps -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/pwd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/readlink -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/realpath -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/reset -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/resume -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/rm -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/rmdir -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sed -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/seq -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setfattr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setpriv -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setserial -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/setsid -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sh -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha1sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha256sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha3sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sha512sum -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/shred -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sleep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sort -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/ssl_client -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/stat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/strings -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/stty -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/sync -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tail -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tar -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tee -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/test -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tftp -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/time -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/top -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/touch -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tr -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tree -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/true -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/truncate -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tsort -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/tty -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/umount -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/uname -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/uniq -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/unlzma -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/unxz -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/unzip -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/usleep -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/vi -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/wc -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/wget -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/which -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xargs -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xxd -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xz -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/xzcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//bin/zcat -> busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/arp -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/blkid -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/blockdev -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/chroot -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/devmem -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/fdisk -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/fsck -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/fsfreeze -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/hwclock -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cdetect -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cdump -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cget -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/i2cset -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/ifconfig -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/insmod -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/ip -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/killall5 -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/loadkmap -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/losetup -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/lsmod -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/mkdosfs -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/mkfs.vfat -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/modinfo -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/ntpd -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/partprobe -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/route -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/seedrng -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/sysctl -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/udhcpc -> ../bin/busybox
  /tmp/tmp.8WKh1F5vTB/bin/..//sbin/vconfig -> ../bin/busybox


--------------------------------------------------
You will probably need to make your busybox binary
setuid root to ensure all configured applets will
work properly.
--------------------------------------------------

make[1]: Leaving directory '/home/user/heads/build/x86/busybox-1.36.1'

Which spams console on every local rebuild when only bash/ash scripts changed. This should vanish, and at the same time, this issue.

tlaurion avatar Apr 19 '24 23:04 tlaurion