Bootstrappable Builds

[tobtoht]

This PR proposes to replace Gitian with Guix to achieve bootstrappable builds for release binaries.

What?

https://youtube.com/watch?v=I2iShmUTEl8

If you have 15 minutes, please consider watching this presentation by Carl Dong linked above as it covers the migration (Gitian → Guix) this PR proposes to make. It explains why reproducibility alone is not enough.

Other resources:

• https://github.com/Sjors/nado-book/blob/master/attacks/guix.md
• https://bootstrappable.org

Why?

• Greater build system security: Bootstrappability allows us to audit and reproduce our toolchain instead of blindly trusting binary downloads. Our build environment can be built from source, all the way down. It allows us to reduce our supply chain attack surface by only including the packages that we need, and nothing else.

• Easier to set up: Guix runs on any Linux distribution and on most architectures (x86_64, aarch64, riscv64). To produce reproducible release binaries, you only need to install Guix and run the build script. This hopefully leads to more participants in the pre-release verified reproduction process.

• More flexibility: Unlike Gitian, we are not limited to the package set of a particular Ubuntu version. Guix allows us to pick and choose our toolchains. This allows us to use the latest compilers while targeting older versions of glibc and would, for example, make it easier to add a modern Rust compiler to our build environment. Packages that are not available in Guix can easily be defined in the manifest or upstreamed.

• Better developer UX: Guix allows us to modify any detail about our build environment with ease. Debugging build issues takes less time because we have shell access to the build environment. The monero source code is bind mounted into the container, so edits to package definitions can be tested incrementally.

• Future proof: Guix is actively developed, Gitian is in maintenance mode (serious bug fixes only).

How to test?

Requirements:

• any Linux distribution
• 50 GB of free disk space
• 4 or more cores recommended
• 2 GB RAM per thread

# Install guix
$ apt install guix  # ubuntu 22.04, debian 11, or later 
or 
# use the install script: https://guix.gnu.org/manual/en/html_node/Binary-Installation.html

# Clone the repo
$ git clone https://github.com/tobtoht/monero
$ cd monero
$ git checkout guix

# Run the builds
$ ./contrib/guix/guix-build  # this will take several hours, uses all available cores
or
$ JOBS=N ./contrib/guix/guix-build # replace N with the desired maximum number of threads

# Obtain a list of hashes
$ uname --machine && find guix/guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum

For more information, see README.md

Notes

Environment	Guix version (NEW)	Gitian version (OLD)
Guix	53396a22af (28 Aug 2024)	n/a
Glibc	2.27	2.27
Binutils	2.38	2.30
GCC	12.4.0	7.5.0
Clang	11.1.0 (darwin, freebsd)	9.0.0
Linux headers	6.1.106 (LTS)*	5.4.0

* this does not affect run-time kernel requirements, see glibc docs.

The guix.yml workflow does not use caching because it uses too much space and causes evictions for other caches (10 GB limit). We only need to run it when there is a change to contrib/{depends,guix}, which happens infrequently and doesn't need to complete quickly.

This PR removes native_clang because Guix provides us with a bootstrapped package and non-guix builds can use the system clang. This PR introduces backwards incompatible compiler flags, bumping the minimum Clang version to 10 (up from 9) for depends builds. Clang >=10 is available in the package managers of all supported distros (including Ubuntu 18.04).

The minimum glibc version for all Linux targets is now 2.27 (Ubuntu 18.04, Debian 10), for more details see https://github.com/monero-project/monero/pull/9171#issuecomment-1952926105.

The source archive now includes all submodules.

To-do

• [ ] All binaries tested

Q&A

What role does Guix serve in the build process?

Guix is used to set-up a reproducible, bootstrappable, containerized build environment. This replaces the Ubuntu 18.04 based environment we used for Gitian builds.

Monero dependencies are built using the depends build system inside the container provided by Guix. While Guix could eventually replace depends, it would be too drastic for this PR.

Can Guix run on any Linux distribution?

Yes, with few exceptions. Installing Guix on an immutable distro like Fedora SilverBlue might not be possible.

To install Guix, use the official install script: https://guix.gnu.org/manual/en/html_node/Binary-Installation.html

Can Guix run on macOS or Windows?

Not natively. You can run Guix builds on macOS inside a virtual machine (even on Apple Silicon). On Windows, it should be possible to install Guix on WSL2.

Can Guix run on aarch64 / riscv64 machines?

Yes. It even produces the same bit-identical release binaries as Guix run on a x86_64 machine. This isn't possible with Gitian.

Targets that don't currently build on non-x86_64: Android, FreeBSD.

Are Guix builds slower than Gitian?

Guix builds are only slower the first time, because it needs to build toolchains (or the entire package graph if --no-substitutes is used) from source. Subsequent runs just as fast (or faster) than Gitian due to caching.

Why Guix instead of Nix?

Nix packages are not bootstrappable.

What does bootstrappable mean in this context?

It means that the package graph for all packages included in our build environment is rooted in a single 357-byte heavily annotated program. We no longer have to trust binaries (downloaded from Ubuntu servers) for reproducible builds.

For more information: https://guix.gnu.org/en/blog/2023/the-full-source-bootstrap-building-from-source-all-the-way-down/

If a new version of Guix is released, does that mean our build environment is no longer reproducible?

No, builds remain reproducible indefinitely, assuming the source code for all packages is archived and remains available. The exact version of Guix, including all of its packages is pinned using time-machine.

More information:

• https://guix.gnu.org/blog/2024/source-code-archiving-in-guix-new-publication/
• https://guix.gnu.org/en/blog/2024/adventures-on-the-quest-for-long-term-reproducible-deployment/

How do we define which packages are included in the build environment?

In manifest.scm. In this file, we can choose packages that are already available in Guix or define packages ourselves.

When should we define packages in the manifest?

If a package, specific package version, or toolchain isn't available in Guix and it isn't possible to upstream or we can't we can't wait on that.

Is the build environment identical for all targets?

No, extra packages can be added for each target.

If we change the time-machine commit, does that mean our build environment changes?

Yes. The manifest generally does not specify which packages versions are included in the build environment. If a package was updated in Guix between the old and new commit, we will have the updated version in our build environment. Updating the time-machine can therefore cause breakage and should be done with caution.

How do I get shell access to the build environment for development purposes?

In contrib/guix/guix-build replace bash -c "cd /monero && bash contrib/guix/libexec/build.sh" with bash.

Then invoke with: FORCE_DIRTY_WORKTREE=1 ./contrib/guix/guix-build.

Where to start review?

contrib/guix/README.md ↓ contrib/guix/guix-build ↓ contrib/guix/manifest.scm ↓ contrib/guix/libexec/build.sh

Troubleshooting

• Might segfault during compilation on first generation Ryzen processors due to a hardware fault.
• Ubuntu 24.04: If you run into mount "none" on "/tmp/guix-directory.": Permission denied you need to create an apparmor profile for Guix, see: https://github.com/monero-project/monero/pull/8929#issuecomment-2325048697

Status

This PR is ready for testing and review.

Target	Builds	Tested
x86_64-linux-gnu	🮱	🮱
arm-linux-gnueabihf	🮱
aarch64-linux-gnu	🮱
riscv64-linux-gnu	🮱
i686-linux-gnu	🮱
i686-w64-mingw32	🮱
x86_64-w64-mingw32	🮱	🮱
x86_64-unknown-freebsd	🮱
x86_64-apple-darwin	🮱
aarch64-apple-darwin	🮱
arm-linux-androideabi	🮱
aarch64-linux-android	🮱	🮱

Testing checklist

• monero-wallet-cli:
  • create a new wallet file
  • open a wallet file that was created with v0.18.3.4
  • receive a transaction
  • send a transaction
  • save and reopen a wallet file
• monerod
  • sync until completion without crashing

Follow-up PRs

• #9207
• #9440

[0xFFFC0000]

I cannot wait until we finalize and merge this. Thanks for your great work @tobtoht

[tobtoht]

I don't expect to make major changes to this PR, it's ready for testing. Reviews on any of the sub-PRs would help move this forward.

[tobtoht]

423dcee4a23f697669b74188588eb1fa023b0921de1ba8c31860c6c670d6cdc2  monero-aarch64-apple-darwin-9ea73c0ff089.tar.bz2
9c40bc1ff4267f25bc2a36649bd5fd06fed2ed0fe1aadfdfca2265248ca78106  monero-aarch64-linux-android-9ea73c0ff089.tar.bz2
d7c1099361828707d2a57e5fb5407f32323bc23c5009942b0c551f17b5fd7f5c  monero-aarch64-linux-gnu-9ea73c0ff089.tar.bz2
486e6282b26dc18c4dbe1f71821bf14bd87352b6eccf29b9753f17d92bad103a  monero-arm-linux-androideabi-9ea73c0ff089.tar.bz2
85c57bde9b139303655c6f57b1305be37e613f51e0f264361b4f6e4136c42dc5  monero-arm-linux-gnueabihf-9ea73c0ff089.tar.bz2
d24844c3396c2f4b3cd0ee0d74a02d26070480361e4fdcf7b3792ac77cbb63dd  monero-i686-linux-gnu-9ea73c0ff089.tar.bz2
14265f23ae8937d233861c3371be32a1a1a6e112c5b5e193ec24f819802f5dc4  monero-i686-w64-mingw32-9ea73c0ff089.zip
49133fe551ca06965b0ace3283fd87a1d1bd4863a5c97e29bf02f892e96feeaa  monero-riscv64-linux-gnu-9ea73c0ff089.tar.bz2
75e452f349dc0996792f3551ff452301c7107080628111f5fd2364fee69e04c6  monero-source-9ea73c0ff089.tar.gz
93c0bd491b284e920cdc042e750fad2b215896f0bcff8736b921769fe75d622f  monero-x86_64-apple-darwin-9ea73c0ff089.tar.bz2
fb9a3725c547d4313577b7937ea1c34a3c4db02eae30159dd1b57e4c447f0a8c  monero-x86_64-linux-gnu-9ea73c0ff089.tar.bz2
e5aca1771d83612844e7e85aeef1d9b7897f0575757cb207e6151ab092073f25  monero-x86_64-unknown-freebsd-9ea73c0ff089.tar.bz2
58d7586c7698e64f20c584d61a98bb4140239c4b6e2184a946a93508355f3650  monero-x86_64-w64-mingw32-9ea73c0ff089.zip

[plowsof]

hashes

423dcee4a23f697669b74188588eb1fa023b0921de1ba8c31860c6c670d6cdc2  monero-aarch64-apple-darwin-9ea73c0ff089.tar.bz2
9c40bc1ff4267f25bc2a36649bd5fd06fed2ed0fe1aadfdfca2265248ca78106  monero-aarch64-linux-android-9ea73c0ff089.tar.bz2
d7c1099361828707d2a57e5fb5407f32323bc23c5009942b0c551f17b5fd7f5c  monero-aarch64-linux-gnu-9ea73c0ff089.tar.bz2
486e6282b26dc18c4dbe1f71821bf14bd87352b6eccf29b9753f17d92bad103a  monero-arm-linux-androideabi-9ea73c0ff089.tar.bz2
85c57bde9b139303655c6f57b1305be37e613f51e0f264361b4f6e4136c42dc5  monero-arm-linux-gnueabihf-9ea73c0ff089.tar.bz2
d24844c3396c2f4b3cd0ee0d74a02d26070480361e4fdcf7b3792ac77cbb63dd  monero-i686-linux-gnu-9ea73c0ff089.tar.bz2
14265f23ae8937d233861c3371be32a1a1a6e112c5b5e193ec24f819802f5dc4  monero-i686-w64-mingw32-9ea73c0ff089.zip
49133fe551ca06965b0ace3283fd87a1d1bd4863a5c97e29bf02f892e96feeaa  monero-riscv64-linux-gnu-9ea73c0ff089.tar.bz2
75e452f349dc0996792f3551ff452301c7107080628111f5fd2364fee69e04c6  monero-source-9ea73c0ff089.tar.gz
93c0bd491b284e920cdc042e750fad2b215896f0bcff8736b921769fe75d622f  monero-x86_64-apple-darwin-9ea73c0ff089.tar.bz2
fb9a3725c547d4313577b7937ea1c34a3c4db02eae30159dd1b57e4c447f0a8c  monero-x86_64-linux-gnu-9ea73c0ff089.tar.bz2
e5aca1771d83612844e7e85aeef1d9b7897f0575757cb207e6151ab092073f25  monero-x86_64-unknown-freebsd-9ea73c0ff089.tar.bz2
58d7586c7698e64f20c584d61a98bb4140239c4b6e2184a946a93508355f3650  monero-x86_64-w64-mingw32-9ea73c0ff089.zip

[kevcrumb]

I recently designed a process for gitian-building entirely in RAM with only 12GB total space required, significantly less than:

• 16GB of free disk space on the partition that /gnu/store will reside in
• 8GB of free disk space per platform triple

Could Guix build results be made to match those generated by gitian?

[tobtoht]

@kevcrumb The figures in the readme are a bit off, I'll update them.

Description	location	All targets	Max. per target
guix cache	/gnu/store	8.0 GB	-
git repository		0.8 GB	-
depends sources cache	SOURCES_PATH or contrib/depends/sources	0.9 GB	-
depends package cache	BASE_CACHE or contrib/depends/built	1.2 GB	290 MB
extracted depends packages	contrib/depends/<target>	6.7 GB	1100 MB
build directories	guix/guix-build-<commit>/build	9.6 GB	980 MB
outputs	guix/guix-build-<commit>/output	1.0 GB	-
total		28.2 GB	13.1 GB

If target specific build/cache directories are removed after each successfully built target, about 13.1 GB of storage is required at any time, not counting a base VM image with guix, git, make and curl installed. I could add an option that does that for storage constrained environments.

[kevcrumb]

about 13.1 GB of storage is required at any time

On a 16 GB system that would leave a mere 2.9 of RAM for the host's OS and the entire build process combined.

Anyway, my question was rather if we could get matching checksums between gitian and Guix, so that both methods could be employed interchangeably.

(dropping the term "reproducible builds" here and a mention of #1854, as I always struggle to find this issue)

[tobtoht]

Anyway, my question was rather if we could get matching checksums between gitian and Guix

No, this is not possible. We can't recreate the Gitian build environment in Guix. Even if we could, it would be unmaintainable and too limiting.

[tobtoht]

$ uname --machine && find guix/guix-build-$(git rev-parse --short=12 HEAD)/output/ -type f -print0 | env LC_ALL=C sort -z | xargs -r0 sha256sum
x86_64
dcf87fdab805f5a0bb77e9656024caed41330c30c3ccc35efd02fbfd3a02dd3a  guix/guix-build-ee3782d83251/output/aarch64-apple-darwin/monero-aarch64-apple-darwin-ee3782d83251.tar.bz2
150cfc02d909ef604eec0a9494538ff8344cf430dec9308a0ab785d028200832  guix/guix-build-ee3782d83251/output/aarch64-linux-gnu/monero-aarch64-linux-gnu-ee3782d83251.tar.bz2
63130e0275501be716181d2c749f8d41d51a9986389243a0a0ab5391c9755c9b  guix/guix-build-ee3782d83251/output/arm-linux-gnueabihf/monero-arm-linux-gnueabihf-ee3782d83251.tar.bz2
e53cdca1094e7059dc2603fcc157b7b9ed9dd753c623259a08e3f80282a1a6a5  guix/guix-build-ee3782d83251/output/dist-archive/monero-source-ee3782d83251.tar.gz
fc58e7c0655d6af309fbdd0e7e3c1507a370767a41428f1b40b99d53ac9b32be  guix/guix-build-ee3782d83251/output/i686-linux-gnu/monero-i686-linux-gnu-ee3782d83251.tar.bz2
12433286439b89b4fa9f4da3565b3ed5394be019653b689c57f9679fe6e904b5  guix/guix-build-ee3782d83251/output/i686-w64-mingw32/monero-i686-w64-mingw32-ee3782d83251.zip
aaeeba15997fbe2e972376e3e8cf3ac171296805d46f84c64e0b92dbd008def5  guix/guix-build-ee3782d83251/output/riscv64-linux-gnu/monero-riscv64-linux-gnu-ee3782d83251.tar.bz2
2b88cd8204c49ebb84e5612ad45c16a7f9d5ddd4acee8323ad27a04b165481d6  guix/guix-build-ee3782d83251/output/x86_64-apple-darwin/monero-x86_64-apple-darwin-ee3782d83251.tar.bz2
d619cf45fbf40b897a2b335ecba4010f3435548ce379ee71bc864024b647264a  guix/guix-build-ee3782d83251/output/x86_64-linux-gnu/monero-x86_64-linux-gnu-ee3782d83251.tar.bz2
04d2fca7317c6b3ba8e3781b3ca21c4adf12f12f31d1d51e1930bb4567622ae1  guix/guix-build-ee3782d83251/output/x86_64-w64-mingw32/monero-x86_64-w64-mingw32-ee3782d83251.zip

aarch64
dcf87fdab805f5a0bb77e9656024caed41330c30c3ccc35efd02fbfd3a02dd3a  guix/guix-build-ee3782d83251/output/aarch64-apple-darwin/monero-aarch64-apple-darwin-ee3782d83251.tar.bz2
150cfc02d909ef604eec0a9494538ff8344cf430dec9308a0ab785d028200832  guix/guix-build-ee3782d83251/output/aarch64-linux-gnu/monero-aarch64-linux-gnu-ee3782d83251.tar.bz2
63130e0275501be716181d2c749f8d41d51a9986389243a0a0ab5391c9755c9b  guix/guix-build-ee3782d83251/output/arm-linux-gnueabihf/monero-arm-linux-gnueabihf-ee3782d83251.tar.bz2
e53cdca1094e7059dc2603fcc157b7b9ed9dd753c623259a08e3f80282a1a6a5  guix/guix-build-ee3782d83251/output/dist-archive/monero-source-ee3782d83251.tar.gz
fc58e7c0655d6af309fbdd0e7e3c1507a370767a41428f1b40b99d53ac9b32be  guix/guix-build-ee3782d83251/output/i686-linux-gnu/monero-i686-linux-gnu-ee3782d83251.tar.bz2
12433286439b89b4fa9f4da3565b3ed5394be019653b689c57f9679fe6e904b5  guix/guix-build-ee3782d83251/output/i686-w64-mingw32/monero-i686-w64-mingw32-ee3782d83251.zip
aaeeba15997fbe2e972376e3e8cf3ac171296805d46f84c64e0b92dbd008def5  guix/guix-build-ee3782d83251/output/riscv64-linux-gnu/monero-riscv64-linux-gnu-ee3782d83251.tar.bz2
2b88cd8204c49ebb84e5612ad45c16a7f9d5ddd4acee8323ad27a04b165481d6  guix/guix-build-ee3782d83251/output/x86_64-apple-darwin/monero-x86_64-apple-darwin-ee3782d83251.tar.bz2
d619cf45fbf40b897a2b335ecba4010f3435548ce379ee71bc864024b647264a  guix/guix-build-ee3782d83251/output/x86_64-linux-gnu/monero-x86_64-linux-gnu-ee3782d83251.tar.bz2
04d2fca7317c6b3ba8e3781b3ca21c4adf12f12f31d1d51e1930bb4567622ae1  guix/guix-build-ee3782d83251/output/x86_64-w64-mingw32/monero-x86_64-w64-mingw32-ee3782d83251.zip

Matching hashes for Linux, Windows, and macOS targets on x86_64 and aarch64 based machines.