go-ethereum icon indicating copy to clipboard operation
go-ethereum copied to clipboard

Published 20 hours ago • verified publisher icon ethereum

Reproducible builds

Open holiman opened this issue 1 year ago • 20 comments

Reproducible builds

This is a little investigation into "do we have reproducible builds in geth?".

A reproducible build means that one can replicate locally a build made on e.g. a build-server. That is, produce an exact matching binary. This is very useful to verify the integrity of the build-servers: any remote machine can be used to watch over the builds.

The Go compiler is, supposedly, reproducible. However, go-ethereum is not pure go

  • It uses a c compiler, thus we need to ensure the same compiler is used
  • It bundles some version-system artefacts at build-time,
  • Maybe other things. To be discovered.

Testing

First, I downloaded the latest build from our downloads-page. The downloads-page lists the checksum as 8d5e138dc3eb7b08cde48966aee0ea79 (note: md5 is not a secure cryptographic hash, but we also provide detached signatures, which offers much better security in verifying integrity).

[user@work go-ethereum]$ md5sum geth-linux-amd64-1.13.13-unstable-fe91d476.tar.gz
8d5e138dc3eb7b08cde48966aee0ea79  geth-linux-amd64-1.13.13-unstable-fe91d476.tar.gz
[user@work go-ethereum]$ md5sum geth-linux-amd64-1.13.13-unstable-fe91d476/geth
1a372833c2a63c95a2f855524eb5fcd9  geth-linux-amd64-1.13.13-unstable-fe91d476/geth

I then tried to create a docker container replicating the enviromment used. Details gleaned from the downloaded file:

$ ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth version
Geth
Version: 1.13.13-unstable
Git Commit: fe91d476ba3e29316b6dc99b6efd4a571481d888
Git Commit Date: 20240213
Architecture: amd64
Go Version: go1.21.6
Operating System: linux
GOPATH=/home/user/go
GOROOT=/usr/local/go

The .travis.yml also gives us some hints:

      dist: bionic
      go: 1.21.x

Dockerfile attempt

Using a dockerfile like this:

from ubuntu:bionic

RUN apt-get update && apt-get install gcc-multilib git ca-certificates wget -yq --no-install-recommends
RUN git clone --branch master https://github.com/ethereum/go-ethereum.git

RUN wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz && \
	rm -rf /usr/local/go && \
	tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz && \
	export PATH=$PATH:/usr/local/go/bin 

RUN cd go-ethereum && git checkout fe91d476ba3e29316b6dc99b6efd4a571481d888 && \
	CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install ./cmd/geth/
RUN md5sum ./build/bin/geth

In order to make the docker-version bundle the git data, we set the TRAVIS,CI env variables. See internal/build/env.go for reasons.

The two builds are not exactly alike in size:

root@208cb9fcfa68:/go-ethereum# ls -l ./build/bin/geth         
-rwxr-xr-x 1 root root 58129760 Feb 14 09:53 ./build/bin/geth

$ ls -la ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth
-rwxr-xr-x 1 user user 58129968 Feb 13 14:55 ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth

Content-wise:

root@208cb9fcfa68:/go-ethereum# strings ./build/bin/geth | head    
/lib64/ld-linux-x86-64.so.2
RAMLiBUAnrbn5zHLQ2v2/WlYmiboMK5ddsyu5qL-z/zajlwZgTLCfStG3HorG6/Utx6Jmui4qlzsokyGBwE
D %$
DD@ 
#@ $
@@	j
k(dB0
0	 b
ljI^
q6-p

VS

$ strings ./geth-linux-amd64-1.13.13-unstable-fe91d476/geth | head
/lib64/ld-linux-x86-64.so.2
JHKPXlVR27nUe4y9sY68/WlYmiboMK5ddsyu5qL-z/zajlwZgTLCfStG3HorG6/QrO6sKmnFVHR7U-WHF3U
x3vo
D %$
DD@
#@ $
@@	j
k(dB0
0	 b
ljI^

holiman avatar Feb 14 '24 09:02 holiman

Actually, ignore reproducing the same build as the travis builder, we don't even reproduce the same build on the same system:

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth 

root@208cb9fcfa68:/go-ethereum# CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth

root@208cb9fcfa68:/go-ethereum# md5sum ./build/bin/geth
1337ffaed216a31fa9a77caf138f642f  ./build/bin/geth

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth 

root@208cb9fcfa68:/go-ethereum# CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth

root@208cb9fcfa68:/go-ethereum# md5sum ./build/bin/geth
4e5180c9678db91d506e223c9a25838a  ./build/bin/geth

holiman avatar Feb 14 '24 11:02 holiman

If we disable the C building, then we get reliable builds on a single machine

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth; CGO_ENABLED=0 CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth; md5sum ./build/bin/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth
9f99056d1537a6f00704e25cc77e8a3f  ./build/bin/geth

root@208cb9fcfa68:/go-ethereum# rm ./build/bin/geth; CGO_ENABLED=0 CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth; md5sum ./build/bin/geth
gotool.go:96: -dlgo version matches active Go version 1.21.6, skipping download.
>>> /usr/local/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /go-ethereum/build/bin/geth ./cmd/geth
9f99056d1537a6f00704e25cc77e8a3f  ./build/bin/geth

holiman avatar Feb 14 '24 11:02 holiman

Got a report that these paths are present in the output:

│ -/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_usbfs.c

[user@work hid]$ go build ./demo.go && strings demo | grep home
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/libusbi.h
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_usbfs.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/events_posix.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_netlink.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/core.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/hotplug.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/io.c
/home/user/go/src/github.com/karalabe/hid/wchar.go
/home/user/go/src/github.com/karalabe/hid/hid_enabled.go
/home/user/go/src/github.com/karalabe/hid/demo.go

[user@work hid]$ go build  -ldflags="-w -s" ./demo.go && strings demo | grep home
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/libusbi.h
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_usbfs.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/events_posix.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/os/linux_netlink.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/core.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/hotplug.c
/home/user/go/src/github.com/karalabe/hid/libusb/libusb/io.c
/home/user/go/src/github.com/karalabe/hid/wchar.go
/home/user/go/src/github.com/karalabe/hid/hid_enabled.go
/home/user/go/src/github.com/karalabe/hid/demo.go


[user@work hid]$ go build -trimpath ./demo.go && strings demo | grep home
[user@work hid]$

This works when imported as a library too

[user@work go-ethereum]$ go build ./cmd/geth  &&  strings ./geth | grep "home/user" | head -n 5
/home/user/go/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/libusbi.h
/home/user/go/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_usbfs.c
/home/user/go/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/events_posix.c
/home/user/go/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_netlink.c
/home/user/go/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/core.c
[user@work go-ethereum]$ go build -trimpath ./cmd/geth  &&  strings ./geth | grep "home/user" | head -n 5
[user@work go-ethereum]$

I don't see these paths in the output binary

[user@work go-ethereum]$ CI=true TRAVIS=true TRAVIS_COMMIT="fe91d476ba3e29316b6dc99b6efd4a571481d888" go run ./build/ci.go install -dlgo ./cmd/geth &&  strings ./build/bin/geth | grep "home/user" | head -n 5
/home/user/.cache/go1.22.2.linux-amd64.tar.gz is up-to-date
>>> /home/user/.cache/geth-go-1.22.2-linux-amd64/go/bin/go build -ldflags "-X github.com/ethereum/go-ethereum/internal/version.gitCommit=fe91d476ba3e29316b6dc99b6efd4a571481d888 -X github.com/ethereum/go-ethereum/internal/version.gitDate=20240213 -extldflags '-Wl,-z,stack-size=0x800000'" -tags urfave_cli_no_docs,ckzg -trimpath -v -o /home/user/go/src/github.com/ethereum/go-ethereum/build/bin/geth ./cmd/geth

holiman avatar Apr 25 '24 13:04 holiman

Hi,

Running this:

wget https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.13.15-c5ba367e.tar.gz
tar -xvf geth-linux-amd64-1.13.15-c5ba367e.tar.gz
cd geth-linux-amd64-1.13.15-c5ba367e
grep -a 'home/travis' geth | strings

I get four occurrences of full Travis paths in the bundle:

/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_netlink.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_usbfs.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/io.c
/home/travis/gopath/pkg/mod/github.com/ethereum/[email protected]/bindings/go/../../src/c_kzg_4844.c

Which are part of the read-only data. readelf -p .rodata geth | grep 'travis'

-trimpath seems to work so perhaps there is something else going on -- looking into it.

Here are some files to reproduce more descriptive diffs using diffoscope.

vivi365 avatar Apr 30 '24 09:04 vivi365

Right. And here's how it looks against a newer binary (1.14.0)

/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/libusbi.h
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/events_posix.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_netlink.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/os/linux_usbfs.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/core.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/hotplug.c
/home/travis/gopath/pkg/mod/github.com/karalabe/[email protected]/libusb/libusb/io.c
/home/travis/gopath/pkg/mod/github.com/ethereum/[email protected]/bindings/go/../../src/c_kzg_4844.c

holiman avatar Apr 30 '24 09:04 holiman

@vivi365 made a great finding here: https://github.com/golang/go/issues/67011, trimpath is broken in ubuntu bionic.

Following that example, I did the same (but with hid, to reduce the build time)

First dockerfile, bionic, 18.04, which is an ESM

FROM ubuntu:bionic
RUN apt-get update && apt-get install gcc-multilib git ca-certificates wget -yq --no-install-recommends
RUN git clone --branch master --depth 1 https://github.com/karalabe/hid
RUN wget https://go.dev/dl/go1.21.6.linux-amd64.tar.gz && \
	rm -rf /usr/local/go && \
	tar -C /usr/local -xzf go1.21.6.linux-amd64.tar.gz && \
	export PATH=$PATH:/usr/local/go/bin

RUN cd hid && CGO_ENABLED=1 /usr/local/go/bin/go build -trimpath ./demo.go
RUN mv /hid/demo /demo && readelf -p .rodata demo | tee 1.txt | grep /hid/libusb | tee 2.txt

results in

#10 0.139   [ 3a745]  /hid/libusb/libusb/libusbi.h
#10 0.139   [ 3a890]  /hid/libusb/libusb/core.c
#10 0.139   [ 3aade]  /hid/libusb/libusb/hotplug.c
#10 0.139   [ 3ab5c]  /hid/libusb/libusb/io.c
#10 0.139   [ 3b9b8]  /hid/libusb/libusb/os/events_posix.c
#10 0.139   [ 3b9e0]  /hid/libusb/libusb/os/linux_netlink.c
#10 0.139   [ 3ba08]  /hid/libusb/libusb/os/linux_usbfs.c

For

  • ubuntu:focal 20.04 LTS, (not yet ESM)
  • ubuntu:jammy 22.04, LTS (not yet ESM)
  • ubuntu:noble 24.04, LTS (not yet ESM)
#9 0.470   [ 3b950]  /_/github.com/karalabe/hid/libusb/libusb/libusbi.h
#9 0.470   [ 3b988]  /_/github.com/karalabe/hid/libusb/libusb/os/events_posix.c
#9 0.470   [ 3b9c8]  /_/github.com/karalabe/hid/libusb/libusb/os/linux_netlink.c
#9 0.470   [ 3ba08]  /_/github.com/karalabe/hid/libusb/libusb/os/linux_usbfs.c
#9 0.470   [ 3ba48]  /_/github.com/karalabe/hid/libusb/libusb/core.c
#9 0.470   [ 3baa8]  /_/github.com/karalabe/hid/libusb/libusb/hotplug.c
#9 0.470   [ 3bae0]  /_/github.com/karalabe/hid/libusb/libusb/io.c

This is good, now it stripped the path /hid/ and all we see is the package-internal paths.

So seems that particular bug is only present in ubuntu. We should bump the CI-builders.

holiman avatar May 06 '24 06:05 holiman

Thanks for bumping to noble, it seems to have fixed the trimpath issue!

And a heads up; the dist being used might be Ubuntu focal, not noble.

  • The GCC version used in geth 1.14.3 is for ubuntu 20.04 focal fossa:
wget https://gethstore.blob.core.windows.net/builds/geth-linux-amd64-1.14.3-ab48ba42.tar.gz
tar -xvf geth-linux-amd64-1.14.3-ab48ba42.tar.gz && cd geth-linux-amd64-1.14.3-ab48ba42
readelf -p .comment geth

String dump of section '.comment':
  [     0]  GCC: (Ubuntu 9.4.0-1ubuntu1~20.04.2) 9.4.0
  • I set up a Travis pipeline which also uses ubuntu noble and checked the os during build.
cat /etc/os-release

NAME="Ubuntu"
VERSION="20.04.6 LTS (Focal Fossa)"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 20.04.6 LTS"
VERSION_ID="20.04"
HOME_URL="https://www.ubuntu.com/"
SUPPORT_URL="https://help.ubuntu.com/"
BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
VERSION_CODENAME=focal
UBUNTU_CODENAME=focal

Might be as they officially do not support noble (yet) https://docs.travis-ci.com/user/reference/linux/.

vivi365 avatar May 13 '24 10:05 vivi365